Nov 04 00:56:29.268611 ip-10-0-129-114 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:56:29.647839 ip-10-0-129-114 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:29.647839 ip-10-0-129-114 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:56:29.647839 ip-10-0-129-114 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:29.647839 ip-10-0-129-114 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:29.651990 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.651883 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.651990 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.651988 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.651990 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.651992 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.651990 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.651995 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.651990 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.651998 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652001 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652004 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652007 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652010 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652014 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652017 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652020 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652022 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652025 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652028 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652031 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652040 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652043 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652046 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652048 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652051 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652054 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652057 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.652167 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652060 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652062 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652065 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652068 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652071 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652073 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652076 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652079 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652081 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652084 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652087 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652089 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652092 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652094 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652097 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652100 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652103 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652106 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652108 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652111 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.652626 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652114 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652117 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652119 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652122 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652124 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652127 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652130 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652132 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652134 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652137 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652140 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652142 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652145 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652147 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652151 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652155 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652158 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652161 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652164 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652166 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.653099 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652169 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652172 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652175 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652180 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652185 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652193 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652197 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652200 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652209 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652212 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652215 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652218 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652221 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652223 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652226 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652228 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652231 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652234 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.653596 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652236 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652239 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652241 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652244 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652268 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652271 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652273 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652276 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652279 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652282 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652284 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652287 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652290 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652292 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652295 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652297 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652300 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652302 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652305 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652310 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.654048 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652316 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652319 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.652328 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.653977 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.653987 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.653991 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.653994 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.653997 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654000 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654002 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654005 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654008 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654010 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654013 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654016 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654018 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654021 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654023 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.654560 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654026 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654029 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654031 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654034 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654037 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654040 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654043 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654045 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654048 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654051 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654055 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654059 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654063 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654066 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654069 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654072 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654075 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654079 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654082 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.655060 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654085 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654088 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654090 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654093 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654096 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654098 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654101 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654103 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654106 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654108 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654111 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654114 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654116 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654119 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654122 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654125 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654128 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654130 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654133 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.655578 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654135 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654138 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654140 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654143 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654145 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654148 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654151 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654154 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654156 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654160 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654162 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654165 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654171 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654173 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654178 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654181 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654183 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654186 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.656040 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654188 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654190 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654193 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654197 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654200 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654203 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654206 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654209 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654211 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654214 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654216 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654219 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654222 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654224 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654227 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654229 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654232 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654235 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654237 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.656575 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654240 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654242 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654262 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654265 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654267 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654270 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654273 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654277 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654280 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654283 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654286 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654288 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654291 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654294 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654834 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654840 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654843 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654846 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654849 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654852 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.657026 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654854 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654857 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654860 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654862 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654865 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654868 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654873 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654875 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654878 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654882 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654885 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654889 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654892 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654896 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654899 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654902 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654906 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654909 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654911 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.657537 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654914 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654917 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654919 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654922 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654924 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654930 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654935 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654938 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654940 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654945 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654948 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654950 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654953 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654955 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654958 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654960 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.657998 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654963 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654965 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654968 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654970 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654973 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654975 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654978 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654981 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654983 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654986 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654989 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654991 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654995 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.654998 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655000 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655003 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655005 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655008 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655012 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655015 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.658475 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655018 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655020 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655024 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655026 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655030 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655033 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655036 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655039 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655041 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655044 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655047 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655049 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655052 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655054 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655057 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655060 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655062 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655065 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655068 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.658944 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655071 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655075 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655078 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655081 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655083 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655086 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655089 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655091 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655094 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655096 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655099 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655101 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655104 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655107 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655109 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655112 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655115 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655117 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655120 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.659406 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655122 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655126 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.655129 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655265 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655273 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655957 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655966 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655971 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655975 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655980 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655984 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655988 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655991 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655995 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.655999 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656002 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656005 2579 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656008 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656011 2579 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656016 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656020 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656026 2579 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656029 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656032 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:56:29.659864 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656036 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656042 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656046 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656050 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656054 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656057 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656061 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656064 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656067 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656071 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656073 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656078 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656082 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656085 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656088 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656092 2579 flags.go:64] FLAG: --enable-server="true" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656095 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656100 2579 flags.go:64] FLAG: --event-burst="100" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656103 2579 flags.go:64] FLAG: --event-qps="50" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656106 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656110 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656113 2579 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656118 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656121 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656124 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:56:29.660449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656127 2579 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656131 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656134 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656138 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656141 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656145 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656148 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656151 2579 flags.go:64] FLAG: --feature-gates="" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656159 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656162 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656166 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656169 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656172 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656176 2579 flags.go:64] FLAG: --help="false" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656179 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656182 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656185 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656188 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656192 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656195 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656198 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656201 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656204 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:56:29.661056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656207 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656210 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656214 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656217 2579 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656220 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656223 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656227 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656229 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656233 2579 flags.go:64] FLAG: --lock-file="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656235 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656238 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656242 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656263 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656267 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656271 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656274 2579 flags.go:64] FLAG: --logging-format="text" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656277 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656282 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656285 2579 flags.go:64] FLAG: --manifest-url="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656289 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656294 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656298 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656302 2579 flags.go:64] FLAG: --max-pods="110" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656305 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656308 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:56:29.661638 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656311 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656314 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656318 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656321 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656324 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656333 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656337 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656340 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656343 2579 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656346 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656349 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656352 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656355 2579 flags.go:64] FLAG: --port="10250" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656359 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656362 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0075e5eec87743520" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656365 2579 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656369 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656372 2579 flags.go:64] FLAG: --register-node="true" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656375 2579 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656379 2579 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656382 2579 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656385 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656389 2579 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656393 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656398 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656402 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:56:29.662302 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656405 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656408 2579 flags.go:64] FLAG: --runonce="false" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656411 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656415 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656418 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656421 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656424 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656427 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656430 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656434 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656437 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656440 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656443 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656446 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656449 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656453 2579 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656455 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656461 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656464 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656467 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656472 2579 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656474 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656477 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656481 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656484 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:56:29.663016 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656487 2579 flags.go:64] FLAG: --v="2" Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656492 2579 flags.go:64] FLAG: --version="false" Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656497 2579 flags.go:64] FLAG: --vmodule="" Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656501 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656505 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656647 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656652 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656655 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656658 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656661 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656664 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656667 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656670 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656673 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656675 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656678 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656681 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656684 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656686 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656690 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.664077 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656694 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656698 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656701 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656703 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656706 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656709 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656712 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656715 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656717 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656720 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656723 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656727 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656730 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656733 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656738 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656740 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656745 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656748 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656751 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.664745 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656753 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656756 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656759 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656761 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656764 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656767 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656770 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656773 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656775 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656778 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656780 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656783 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656785 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656788 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656791 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656793 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656796 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656799 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656802 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.665222 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656804 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656807 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656809 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656812 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656816 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656820 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656824 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656827 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656830 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656833 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656838 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656842 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656845 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656847 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656850 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656853 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656855 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656858 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656861 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.665713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656863 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656866 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656868 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656871 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656874 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656879 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656881 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656884 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656887 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656889 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656892 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656894 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656897 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656900 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656902 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656905 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656909 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656914 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.666214 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656917 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656919 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656922 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656926 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656929 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656934 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656937 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656940 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656942 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656945 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656947 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.656950 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.656982 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657026 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657032 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.666713 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657035 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657042 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657045 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657047 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657050 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657053 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657056 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657061 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657064 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657067 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657069 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657073 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657076 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657079 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657082 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657084 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657087 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657090 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.667106 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657093 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657097 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657100 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657103 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657106 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657108 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657111 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657114 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657116 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657119 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657123 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657126 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657129 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657132 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657135 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657138 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657141 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657144 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657146 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.667697 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657149 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657152 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657154 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657157 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657160 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657162 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657166 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657169 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657171 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657174 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657176 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657179 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657182 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657188 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657190 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657193 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657196 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657199 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657202 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.668151 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657205 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657207 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657210 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657213 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657215 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657218 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657221 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657223 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657226 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657228 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657231 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657234 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657236 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657239 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657241 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657244 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657258 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657262 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657266 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.668621 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657272 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657275 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657277 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657280 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657283 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657286 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657288 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657293 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657296 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657305 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657309 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657313 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657317 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657319 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657322 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657325 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657327 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657330 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657333 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.669069 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657335 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657338 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657340 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657344 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657346 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657349 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657351 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.657354 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.665712 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.665736 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.666728 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.666735 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666810 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666816 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666820 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666823 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666826 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666829 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666831 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666835 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.669607 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666837 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666840 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666843 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666845 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666848 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666851 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666854 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666856 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666859 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666862 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666865 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666868 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666871 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666874 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666876 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666880 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666883 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666885 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666888 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.670079 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666890 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666894 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666898 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666901 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666903 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666906 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666909 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666914 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666917 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666920 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666922 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666925 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666928 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666931 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666933 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666936 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666938 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666941 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666944 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.670579 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666946 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666949 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666951 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666954 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666958 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666963 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666966 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666968 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666971 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666974 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666977 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666979 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666982 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666985 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666988 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666991 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666994 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.666997 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667000 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.671062 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667003 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667006 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667008 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667011 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667014 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667016 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667019 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667022 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667025 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667028 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667031 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667034 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667037 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667041 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667044 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667047 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667049 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667052 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667055 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.671660 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667058 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667060 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667063 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667065 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667068 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667071 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667073 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667076 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667079 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667082 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667085 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667087 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667091 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667095 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667098 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667101 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667104 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667106 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667109 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667113 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.672155 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667116 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.667165 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667336 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667339 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667344 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667357 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667361 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667364 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667367 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667370 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667373 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667376 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667379 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.672683 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667381 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667384 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667387 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667390 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667392 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667395 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667398 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667401 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667404 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667406 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667409 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667412 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667415 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667418 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667420 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667423 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667425 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667428 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667430 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667433 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.673082 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667435 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667438 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667441 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667444 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667446 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667450 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667452 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667455 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667458 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667460 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667463 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667465 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667468 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667471 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667473 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667476 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667478 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667482 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667485 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667487 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.673750 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667490 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667493 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667496 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667498 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667501 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667504 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667506 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667508 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667511 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667513 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667516 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667519 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667522 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667525 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667527 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667530 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667533 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667535 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667539 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667541 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.674573 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667544 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667546 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667549 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667551 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667554 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667556 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667559 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667561 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667564 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667566 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667569 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667572 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667574 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667577 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667580 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667584 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667587 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667590 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667593 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.675188 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667595 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667598 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667600 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667603 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667605 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667607 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667610 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667613 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667615 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667618 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667620 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667623 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667626 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667629 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.667654 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667689 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:29.675747 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667693 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667696 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667699 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667702 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667705 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667708 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667710 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667713 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667716 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667718 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667721 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667724 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667729 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667732 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667735 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667737 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667743 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667745 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:29.676136 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667749 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667753 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667756 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667759 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667761 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667764 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667770 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667773 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667775 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667779 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667781 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667784 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667787 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667789 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667792 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667794 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667797 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667800 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667802 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:29.676816 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667808 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667811 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667813 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667816 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667820 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667822 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667825 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667827 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667830 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667832 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667835 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667838 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667841 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667843 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667846 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667848 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667851 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667853 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667856 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:29.677360 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667858 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667861 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667863 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667867 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667872 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667875 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667877 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667880 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667882 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667885 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667888 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667891 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667894 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667896 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667899 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667901 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667905 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667908 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667910 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:29.677844 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667913 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667916 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667918 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667921 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667925 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667928 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667931 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667933 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667936 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667939 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667942 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667944 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667947 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667949 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667952 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667955 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667958 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667964 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667966 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:29.678396 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667969 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667972 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667974 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667977 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667979 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:29.667985 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.668059 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.670188 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.671025 2579 server.go:1014] "Starting client certificate rotation" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.671152 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.671242 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.672932 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:29.679057 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.674116 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:29.685655 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.685630 2579 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:56:29.688907 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.688889 2579 log.go:25] "Validated CRI v1 image API" Nov 04 00:56:29.689819 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.689796 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:56:29.696501 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.696466 2579 fs.go:135] Filesystem UUIDs: map[102c7c3d-248f-4798-acbf-0e6ed155630a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc83f719-d469-4699-bfca-9d4d0de43477:/dev/nvme0n1p3] Nov 04 00:56:29.696501 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.696497 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 00:56:29.703374 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.703198 2579 manager.go:217] Machine: {Timestamp:2026-11-04 00:56:29.701797198 +0000 UTC m=+0.317560043 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100753 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dcbb1326988890659d34018b66fbe SystemUUID:ec2dcbb1-3269-8889-0659-d34018b66fbe BootID:9cb3c6d0-89ba-4a68-9592-92117f221526 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dc:0f:7d:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dc:0f:7d:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:5c:d7:3e:fb:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:56:29.703374 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.703346 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:56:29.703564 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.703450 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:56:29.705594 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705562 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:56:29.705737 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705596 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-114.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:56:29.705816 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705739 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:56:29.705816 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705748 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:56:29.705816 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705787 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:29.705816 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.705807 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:29.706565 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.706554 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:56:29.709876 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.709859 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:56:29.710028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710018 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:56:29.710567 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710554 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:56:29.710599 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710572 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:56:29.710599 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710585 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:56:29.710599 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710595 2579 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:56:29.710694 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710604 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:56:29.711005 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.710985 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:29.716159 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.716127 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:56:29.721513 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.721494 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:29.722330 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722315 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:56:29.722369 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722348 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:56:29.722562 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722548 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:56:29.722600 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722577 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:56:29.722600 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722592 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:56:29.722662 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722605 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:56:29.722662 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722618 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:56:29.722662 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.722630 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:56:29.723244 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723234 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:56:29.723288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723245 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:56:29.723288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723266 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:56:29.723288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723272 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:56:29.723288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723282 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:56:29.723479 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.723291 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:56:29.724641 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.724627 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:56:29.724688 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.724642 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:56:29.728900 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.728878 2579 server.go:1263] "Started kubelet" Nov 04 00:56:29.729025 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.728904 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:56:29.729698 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.729629 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:56:29.729773 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.729725 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:56:29.729934 ip-10-0-129-114 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:56:29.731998 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.731975 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:56:29.733378 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.733362 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:56:29.737174 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.737155 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:56:29.737266 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.737168 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:56:29.737433 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.737407 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:56:29.738000 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.737981 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:56:29.738000 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.737985 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:56:29.738125 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738034 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:56:29.738265 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738237 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:56:29.738265 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738265 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:56:29.738376 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.738357 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 00:56:29.738425 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738378 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:56:29.738425 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738393 2579 factory.go:55] Registering systemd factory Nov 04 00:56:29.738425 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738402 2579 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:56:29.738683 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738671 2579 factory.go:153] Registering CRI-O factory Nov 04 00:56:29.738683 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738685 2579 factory.go:223] Registration of the crio container factory successfully Nov 04 00:56:29.738766 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738698 2579 factory.go:103] Registering Raw factory Nov 04 00:56:29.738766 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.738710 2579 manager.go:1199] Started watching for new ooms in manager Nov 04 00:56:29.741285 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.741241 2579 manager.go:319] Starting recovery of all containers Nov 04 00:56:29.751680 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.751657 2579 manager.go:324] Recovery completed Nov 04 00:56:29.757378 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.757356 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.759738 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.759719 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.759829 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.759753 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.759829 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.759768 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.760376 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.760355 2579 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:56:29.760477 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.760374 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:56:29.760477 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.760388 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.760477 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.760436 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:56:29.761145 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761130 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.761222 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761156 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.761222 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761167 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.761709 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761693 2579 policy_none.go:50] "Start" Nov 04 00:56:29.761780 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761714 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:56:29.761780 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.761728 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:56:29.762894 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.762881 2579 policy_none.go:44] "Start" Nov 04 00:56:29.816321 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.816045 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:56:29.822529 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.822511 2579 manager.go:341] "Starting Device Plugin manager" Nov 04 00:56:29.822652 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.822550 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:56:29.822652 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.822561 2579 server.go:87] "Starting device plugin registration server" Nov 04 00:56:29.822872 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.822859 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:56:29.822934 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.822873 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:56:29.823065 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.823048 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:56:29.823178 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.823158 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:56:29.823178 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.823174 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:56:29.823694 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.823671 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:56:29.823757 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.823730 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:56:29.871986 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.871952 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-114.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:56:29.873464 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.872023 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-114.us-west-2.compute.internal.18e4b1916b48af3b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-114.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-114.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:29.728599867 +0000 UTC m=+0.344362710,LastTimestamp:2026-11-04 00:56:29.728599867 +0000 UTC m=+0.344362710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-114.us-west-2.compute.internal,}" Nov 04 00:56:29.873753 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.873721 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:56:29.875051 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.875027 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:56:29.875051 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.875050 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:56:29.875206 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.875105 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:56:29.875206 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.875142 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 00:56:29.921921 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:29.921844 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-114.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 00:56:29.923001 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.922989 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.924021 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.924005 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.924117 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.924034 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.924117 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.924055 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.924117 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.924086 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:29.949094 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.949054 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:56:29.949233 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.949140 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7c5l5" Nov 04 00:56:29.975838 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.975773 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal"] Nov 04 00:56:29.976018 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.975874 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.976943 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.976924 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.977056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.976960 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.977056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.976984 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.978671 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.978655 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.978836 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.978819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:29.978886 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.978854 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.979534 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979514 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.979610 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979550 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.979610 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979551 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.979610 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979563 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.979610 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979589 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.979610 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.979603 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:29.981237 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.981219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:29.981354 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.981268 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:29.982112 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.982093 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:29.982192 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.982129 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:29.982192 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:29.982145 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:30.007126 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:30.007105 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.009899 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:30.009874 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.017603 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.017572 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:30.019218 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.019201 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:30.039574 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.039518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.039704 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.039582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.039704 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.039627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.109892 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.109861 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:30.110453 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.110435 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:30.111498 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.111478 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.111585 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:30.111507 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-114.us-west-2.compute.internal\": node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:56:30.138557 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.138529 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.139867 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.139847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.139953 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.139901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.139953 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.139926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.139953 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.139933 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.140075 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.139982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.140075 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.140013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.198391 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.198311 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:56:30.199019 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.199004 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:30.256299 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:30.256236 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:56:31.380739 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.380687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:31.383456 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.383420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:56:31.742200 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.742109 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 00:56:31.742822 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.742324 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:56:31.742822 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.742353 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:56:31.742822 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:31.742290 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ab4f6215569114d8cb596808b4941cb0-c1f4c936e9dcd2c2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-114.us-west-2.compute.internal.18e4b1916d23cb2d\": read tcp 10.0.129.114:36530->3.131.145.49:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-114.us-west-2.compute.internal.18e4b1916d23cb2d default 5234 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-114.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-114.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-114.us-west-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:29 +0000 UTC,LastTimestamp:2026-11-04 00:56:29.976942243 +0000 UTC m=+0.592705093,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-114.us-west-2.compute.internal,}" Nov 04 00:56:31.782106 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.782068 2579 apiserver.go:52] "Watching apiserver" Nov 04 00:56:31.808350 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:31.808323 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 00:56:32.024226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.024127 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:30 +0000 UTC" deadline="2028-06-18 02:35:02.989520229 +0000 UTC" Nov 04 00:56:32.024226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.024165 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14209h38m30.965357999s" Nov 04 00:56:32.030115 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.030094 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:56:32.030388 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.030371 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-t5scf","openshift-multus/multus-additional-cni-plugins-g69bf","openshift-multus/network-metrics-daemon-slvp5","openshift-network-diagnostics/network-check-target-56frc","openshift-network-operator/iptables-alerter-zs49x","openshift-ovn-kubernetes/ovnkube-node-9dx8l","kube-system/konnectivity-agent-8knts","openshift-cluster-node-tuning-operator/tuned-bkc2z","openshift-dns/node-resolver-7g22g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b","openshift-multus/multus-ls9l7","kube-system/global-pull-secret-syncer-lkssf","kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal"] Nov 04 00:56:32.033033 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.032999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.035685 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.035662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.037533 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.037516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.037620 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.037580 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:32.039449 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.039427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:32.039534 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.039514 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:32.041392 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.041376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.043525 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.043498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.045640 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.045619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.047829 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.047811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.049924 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.049904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.052223 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.052205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.054331 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.054314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.056486 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.056464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.056585 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.056538 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:32.082129 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.082102 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.082129 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.082117 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:56:32.082303 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.082124 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.082303 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.082122 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gczmn\"" Nov 04 00:56:32.084028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.083971 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:56:32.084028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.084007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.084028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.084010 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.084028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.084007 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:56:32.084028 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.084017 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:56:32.084353 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.084114 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gr6q\"" Nov 04 00:56:32.089364 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.089336 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:56:32.089533 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.089514 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql7fx\"" Nov 04 00:56:32.089614 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.089545 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.089614 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.089597 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.091458 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.091437 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:56:32.091910 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.091889 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.092025 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.091917 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.092025 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.091950 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:56:32.092025 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.092002 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d9rj\"" Nov 04 00:56:32.092025 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.092012 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:56:32.092179 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.092115 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:56:32.110694 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.110657 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:56:32.115520 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.115670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.115670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.115670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn8f6\" (UniqueName: \"kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.115670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.115670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.115830 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.115830 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:32.115830 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.115830 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.115830 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.115968 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.115968 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.115968 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.115968 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.116109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.115969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.116109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.116109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.116109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.116245 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrp25\" (UniqueName: \"kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.116245 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.116245 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116245 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.116245 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.116379 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.116379 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg2xd\" (UniqueName: \"kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.116379 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116379 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116379 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpm9d\" (UniqueName: \"kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.116556 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.116741 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.116741 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tsl\" (UniqueName: \"kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.116741 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116825 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116825 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116825 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116912 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.116912 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdtm\" (UniqueName: \"kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.117011 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.117011 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117011 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.116995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117149 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117149 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117149 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.117149 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.117321 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117353 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117395 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.117395 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.117480 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.117480 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.117562 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlntm\" (UniqueName: \"kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.117562 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.117562 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxcw\" (UniqueName: \"kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.117562 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.117712 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.117712 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.117712 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.117855 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.117855 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcs4\" (UniqueName: \"kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.117855 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.117855 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.118010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.118010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.118010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.118010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.118010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.117975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.118176 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.118176 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.118176 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.118176 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.118176 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.118372 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.118372 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.118225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.124771 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.124745 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:56:32.124771 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.124747 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tdv96\"" Nov 04 00:56:32.124771 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.124746 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:56:32.124771 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.124757 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.127085 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.127056 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.127191 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.127060 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:56:32.127191 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.127106 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kn9tc" Nov 04 00:56:32.127191 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.127146 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.128165 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128144 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:56:32.128262 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128210 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:56:32.128262 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128216 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.128262 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128259 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:56:32.128386 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128264 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6tf45\"" Nov 04 00:56:32.130931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128669 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2w2b9\"" Nov 04 00:56:32.130931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128676 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:56:32.130931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128836 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qxmlz\"" Nov 04 00:56:32.130931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.128848 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-66h22\"" Nov 04 00:56:32.219040 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219040 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpm9d\" (UniqueName: \"kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.219264 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.219263 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.219347 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:32.719323771 +0000 UTC m=+2.264353111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tsl\" (UniqueName: \"kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219447 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219556 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.219669 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdtm\" (UniqueName: \"kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219745 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219781 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219781 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.219965 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220050 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.220346 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.220115 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220115 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.220168 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:56:32.72015596 +0000 UTC m=+2.265185280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220230 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlntm\" (UniqueName: \"kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221128 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxcw\" (UniqueName: \"kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220296 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220428 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220449 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcs4\" (UniqueName: \"kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.221919 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220637 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220694 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220870 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.222746 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.220968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8f6\" (UniqueName: \"kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.223593 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221566 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrp25\" (UniqueName: \"kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.224409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2xd\" (UniqueName: \"kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.221977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222024 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.225226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.222877 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.223827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.223915 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.223926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.223972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.224088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.226068 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.224180 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.274786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.274715 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:32.274786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.274747 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:32.274976 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.274825 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:32.274976 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.274928 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:32.774905993 +0000 UTC m=+2.319935314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:32.275486 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.275455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdtm\" (UniqueName: \"kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.275617 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.275504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcs4\" (UniqueName: \"kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.275617 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.275458 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tsl\" (UniqueName: \"kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.276069 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.276043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxcw\" (UniqueName: \"kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.276165 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.276027 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlntm\" (UniqueName: \"kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.276439 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.276424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8f6\" (UniqueName: \"kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.277255 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.277234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2xd\" (UniqueName: \"kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.277586 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.277570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrp25\" (UniqueName: \"kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.285329 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.285300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpm9d\" (UniqueName: \"kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.368824 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.368790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:56:32.396885 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.396849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:56:32.404303 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.404277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:56:32.438769 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.438733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:32.438957 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.438855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:32.457668 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.457421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:56:32.476859 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.476825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7g22g" Nov 04 00:56:32.498973 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.498946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:56:32.501789 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.501100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ls9l7" Nov 04 00:56:32.562200 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:32.562158 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a400870bd799b9a4f8d0b646d2dfe4c.slice/crio-23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37 WatchSource:0}: Error finding container 23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37: Status 404 returned error can't find the container with id 23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37 Nov 04 00:56:32.564182 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:32.564153 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ea24cc66811a8502960a15b9987f67.slice/crio-48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533 WatchSource:0}: Error finding container 48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533: Status 404 returned error can't find the container with id 48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533 Nov 04 00:56:32.569292 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.569271 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:56:32.725577 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.725537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:32.725843 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.725585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:32.725843 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.725689 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:32.725843 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.725705 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:32.725843 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.725754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:33.725739069 +0000 UTC m=+3.270768409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:32.725843 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.725768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:56:33.725761939 +0000 UTC m=+3.270791256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:32.826731 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.826640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:32.827406 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.826775 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:32.827406 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.826794 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:32.827406 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.826805 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:32.827406 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:32.826858 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:33.826843131 +0000 UTC m=+3.371872461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:32.952282 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.952227 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"93ea24cc66811a8502960a15b9987f67","Type":"ContainerStarted","Data":"48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533"} Nov 04 00:56:32.953184 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:32.953160 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerStarted","Data":"23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37"} Nov 04 00:56:33.130777 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.130690 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:31 +0000 UTC" deadline="2028-07-14 23:46:46.402226678 +0000 UTC" Nov 04 00:56:33.130777 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.130718 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14854h50m13.271511925s" Nov 04 00:56:33.267897 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.267857 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d17845_e07f_43a7_b2bf_c73d741f180f.slice/crio-244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f WatchSource:0}: Error finding container 244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f: Status 404 returned error can't find the container with id 244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f Nov 04 00:56:33.270172 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.270151 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:33.276184 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.276159 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:33.287550 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.287514 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858fcc1f_21e9_4e03_b22e_115a8377a8fc.slice/crio-4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247 WatchSource:0}: Error finding container 4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247: Status 404 returned error can't find the container with id 4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247 Nov 04 00:56:33.308746 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.308718 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7323316_cacf_4ca1_afb1_7b38b203dfd7.slice/crio-d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5 WatchSource:0}: Error finding container d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5: Status 404 returned error can't find the container with id d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5 Nov 04 00:56:33.352372 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.352333 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692caf64_a543_4631_83a9_ce22eb1ec34a.slice/crio-078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97 WatchSource:0}: Error finding container 078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97: Status 404 returned error can't find the container with id 078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97 Nov 04 00:56:33.402083 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.402049 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46765500_9cd2_4c86_a0cb_046a98b6ea69.slice/crio-7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a WatchSource:0}: Error finding container 7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a: Status 404 returned error can't find the container with id 7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a Nov 04 00:56:33.402848 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.402488 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9554504_e458_45eb_ac2f_fb8bd160802e.slice/crio-74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a WatchSource:0}: Error finding container 74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a: Status 404 returned error can't find the container with id 74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a Nov 04 00:56:33.409191 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.409167 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76833e6_9f22_425e_b95f_c8989faf3e8b.slice/crio-7b894f8a21bf4efca72017e5b0e62aa2b782821ff9fb518e4e217f84bc2a3917 WatchSource:0}: Error finding container 7b894f8a21bf4efca72017e5b0e62aa2b782821ff9fb518e4e217f84bc2a3917: Status 404 returned error can't find the container with id 7b894f8a21bf4efca72017e5b0e62aa2b782821ff9fb518e4e217f84bc2a3917 Nov 04 00:56:33.459915 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:56:33.459884 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d094ba_dfa9_48bc_8ae7_028003bae8a8.slice/crio-f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2 WatchSource:0}: Error finding container f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2: Status 404 returned error can't find the container with id f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2 Nov 04 00:56:33.733565 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.733468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:33.733565 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.733534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:33.733786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.733656 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:33.733786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.733709 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:33.733786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.733739 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:35.73371772 +0000 UTC m=+5.278747051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:33.733786 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.733763 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:56:35.733749067 +0000 UTC m=+5.278778398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:33.834696 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.834652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:33.835263 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.834823 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:33.835263 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.834840 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:33.835263 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.834852 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:33.835263 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.834916 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:35.834898927 +0000 UTC m=+5.379928245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:33.946321 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.946278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:33.946321 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.946323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:33.946568 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.946413 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:33.946568 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.946488 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:33.946568 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.946553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:33.946677 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:33.946628 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:33.956022 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.955968 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8knts" event={"ID":"f9554504-e458-45eb-ac2f-fb8bd160802e","Type":"ContainerStarted","Data":"74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a"} Nov 04 00:56:33.957586 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.957549 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5"} Nov 04 00:56:33.959204 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.959175 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zs49x" event={"ID":"858fcc1f-21e9-4e03-b22e-115a8377a8fc","Type":"ContainerStarted","Data":"4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247"} Nov 04 00:56:33.960755 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.960727 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5scf" event={"ID":"33d17845-e07f-43a7-b2bf-c73d741f180f","Type":"ContainerStarted","Data":"244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f"} Nov 04 00:56:33.962416 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.962366 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ls9l7" event={"ID":"75d094ba-dfa9-48bc-8ae7-028003bae8a8","Type":"ContainerStarted","Data":"f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2"} Nov 04 00:56:33.963553 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.963520 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" event={"ID":"d76833e6-9f22-425e-b95f-c8989faf3e8b","Type":"ContainerStarted","Data":"7b894f8a21bf4efca72017e5b0e62aa2b782821ff9fb518e4e217f84bc2a3917"} Nov 04 00:56:33.965400 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.965375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" event={"ID":"46765500-9cd2-4c86-a0cb-046a98b6ea69","Type":"ContainerStarted","Data":"7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a"} Nov 04 00:56:33.966825 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:33.966802 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7g22g" event={"ID":"692caf64-a543-4631-83a9-ce22eb1ec34a","Type":"ContainerStarted","Data":"078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97"} Nov 04 00:56:34.131384 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:34.131292 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:31 +0000 UTC" deadline="2028-07-20 14:44:30.776997904 +0000 UTC" Nov 04 00:56:34.131384 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:34.131333 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14989h47m56.645670037s" Nov 04 00:56:34.983069 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:34.982961 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerStarted","Data":"461614a38d17871fba18f370edd54bc112b85cc54e338e1cfafdaf73b9104750"} Nov 04 00:56:35.755353 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.755245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:35.755353 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.755308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:35.755568 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.755482 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:35.755568 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.755547 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:56:39.755529586 +0000 UTC m=+9.300558930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:35.755717 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.755612 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:35.755717 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.755645 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:39.755634276 +0000 UTC m=+9.300663610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:35.856575 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.856514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:35.856796 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.856743 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:35.856796 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.856762 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:35.856796 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.856774 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:35.856928 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.856834 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:39.856814165 +0000 UTC m=+9.401843485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:35.946918 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.946851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:35.947109 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.947001 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:35.947109 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.947089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:35.947228 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.947172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:35.947326 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:35.947236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:35.947326 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:35.947305 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:37.011907 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.011631 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"93ea24cc66811a8502960a15b9987f67","Type":"ContainerStarted","Data":"405b4e60fcdf19e8e146e6fff7f55222d8043b60ae6fe17876d275c908d3a4c5"} Nov 04 00:56:37.024327 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.024288 2579 generic.go:356] "Generic (PLEG): container finished" podID="5a400870bd799b9a4f8d0b646d2dfe4c" containerID="af8bdba16fa2ee5dc98d9540c0ee56307ff158c05b56dc8eb5903a46d68f63c2" exitCode=0 Nov 04 00:56:37.024550 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.024351 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerDied","Data":"af8bdba16fa2ee5dc98d9540c0ee56307ff158c05b56dc8eb5903a46d68f63c2"} Nov 04 00:56:37.223204 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.223142 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" podStartSLOduration=6.223119337 podStartE2EDuration="6.223119337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:37.118592264 +0000 UTC m=+6.663621604" watchObservedRunningTime="2026-11-04 00:56:37.223119337 +0000 UTC m=+6.768148675" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.946139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:37.946271 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.946333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:37.946401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:37.946449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:37.946579 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:37.946527 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:38.042932 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:38.042887 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerStarted","Data":"1b12e4d4cb71720ffdb0e4b7830481b53a1610d11bfa2ca822410acccca6f632"} Nov 04 00:56:38.152558 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:38.152365 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" podStartSLOduration=7.152344711 podStartE2EDuration="7.152344711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:38.15135461 +0000 UTC m=+7.696383950" watchObservedRunningTime="2026-11-04 00:56:38.152344711 +0000 UTC m=+7.697374052" Nov 04 00:56:39.050817 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.050175 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8knts" event={"ID":"f9554504-e458-45eb-ac2f-fb8bd160802e","Type":"ContainerStarted","Data":"ee2773262a966922bb1040503e381aeb0a3d0a5f4f15b4b3f902399bc6601041"} Nov 04 00:56:39.054285 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.053436 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="a188bed8ad3576acba6c894a0453d45ea78d10f6dc81fca048471e03013da2a9" exitCode=0 Nov 04 00:56:39.054285 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.054224 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"a188bed8ad3576acba6c894a0453d45ea78d10f6dc81fca048471e03013da2a9"} Nov 04 00:56:39.156359 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.155637 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8knts" podStartSLOduration=3.63515788 podStartE2EDuration="8.155619373s" totalImagesPullingTime="4.520461493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:39.154743739 +0000 UTC m=+8.699773287" watchObservedRunningTime="2026-11-04 00:56:39.155619373 +0000 UTC m=+8.700648713" Nov 04 00:56:39.794595 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.794561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:39.794762 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.794657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:39.794827 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.794813 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:39.794891 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.794881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:47.794863577 +0000 UTC m=+17.339892900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:39.795288 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.795272 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:39.795394 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.795319 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:56:47.795306531 +0000 UTC m=+17.340335853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:39.896583 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.895903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:39.896583 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.896109 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:39.896583 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.896129 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:39.896583 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.896154 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:39.896583 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.896212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:56:47.896193307 +0000 UTC m=+17.441222629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:39.947103 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.947045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:39.947286 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.947134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:39.947286 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.947264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:39.947411 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:39.947302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:39.947460 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.947420 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:39.947538 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:39.947512 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:40.059167 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:40.059130 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5scf" event={"ID":"33d17845-e07f-43a7-b2bf-c73d741f180f","Type":"ContainerStarted","Data":"7529a3cf6664ed60fbaf63a4a6191eb650296b13732126d9c72d30620abbc1cb"} Nov 04 00:56:40.062399 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:40.062329 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" event={"ID":"46765500-9cd2-4c86-a0cb-046a98b6ea69","Type":"ContainerStarted","Data":"dedd3af31892bb788dad80ac05d856b575e95ab8b0de1642c468b5af7d91e323"} Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:41.946219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:41.946443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:41.946557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:41.946651 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:41.946705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:41.946813 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:41.946774 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:42.043083 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.042976 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:56:42.080056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.080001 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ls9l7" event={"ID":"75d094ba-dfa9-48bc-8ae7-028003bae8a8","Type":"ContainerStarted","Data":"d2f700a4152bdb3becbcffff03d72ebecb1ab2bd2b019c9cd7d3be71d56c8885"} Nov 04 00:56:42.087037 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.086954 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" event={"ID":"46765500-9cd2-4c86-a0cb-046a98b6ea69","Type":"ContainerStarted","Data":"92e238b6810b5b118f7d6b1b6d311cadd9a5249b9daeff0173645517295b768e"} Nov 04 00:56:42.187156 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.187101 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ls9l7" podStartSLOduration=3.18862691 podStartE2EDuration="11.18708738s" totalImagesPullingTime="7.99846047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:42.186229291 +0000 UTC m=+11.731258653" watchObservedRunningTime="2026-11-04 00:56:42.18708738 +0000 UTC m=+11.732116719" Nov 04 00:56:42.187337 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.187287 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t5scf" podStartSLOduration=5.579541091 podStartE2EDuration="11.187275843s" totalImagesPullingTime="5.607734752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:40.163931506 +0000 UTC m=+9.708960846" watchObservedRunningTime="2026-11-04 00:56:42.187275843 +0000 UTC m=+11.732305185" Nov 04 00:56:42.899931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.899649 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:56:42.043082614Z","UUID":"a99bad10-9c2a-4c0b-8af1-2f937835a276","Handler":null,"Name":"","Endpoint":""} Nov 04 00:56:42.901410 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.901388 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:56:42.901538 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.901440 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:56:42.917340 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.917106 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:42.918288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:42.918266 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:43.092621 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.092271 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerStarted","Data":"0ecd94ee76e7b75f1297145203dc332041ae67fb16f01093b4bb0f335dd26b9d"} Nov 04 00:56:43.094135 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.094104 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7g22g" event={"ID":"692caf64-a543-4631-83a9-ce22eb1ec34a","Type":"ContainerStarted","Data":"6929562ebd841e3efeb06b72c877aee3e262f8f091c05c1070c59effdeada036"} Nov 04 00:56:43.306909 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.306843 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7g22g" podStartSLOduration=3.61461673 podStartE2EDuration="12.306826087s" totalImagesPullingTime="8.692209357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:43.306237927 +0000 UTC m=+12.851267248" watchObservedRunningTime="2026-11-04 00:56:43.306826087 +0000 UTC m=+12.851855426" Nov 04 00:56:43.946422 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.946394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:43.946422 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.946417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:43.946684 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:43.946504 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:43.946684 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:43.946530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:43.946684 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:43.946598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:43.946684 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:43.946673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:44.097557 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:44.097524 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zs49x" event={"ID":"858fcc1f-21e9-4e03-b22e-115a8377a8fc","Type":"ContainerStarted","Data":"6812620f1324314ced63da675bcb91a2146eee522067bcf88c6988ecda4d73d7"} Nov 04 00:56:44.099705 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:44.099679 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="0ecd94ee76e7b75f1297145203dc332041ae67fb16f01093b4bb0f335dd26b9d" exitCode=0 Nov 04 00:56:44.099816 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:44.099732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"0ecd94ee76e7b75f1297145203dc332041ae67fb16f01093b4bb0f335dd26b9d"} Nov 04 00:56:44.205000 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:44.204905 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zs49x" podStartSLOduration=4.785770359 podStartE2EDuration="13.204891818s" totalImagesPullingTime="8.419121459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:44.204492881 +0000 UTC m=+13.749522219" watchObservedRunningTime="2026-11-04 00:56:44.204891818 +0000 UTC m=+13.749921156" Nov 04 00:56:45.103405 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.103370 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" event={"ID":"d76833e6-9f22-425e-b95f-c8989faf3e8b","Type":"ContainerStarted","Data":"c1670c15388e03bc9c833afbe0c75b97004901a2b4442acb054b1071a474f71e"} Nov 04 00:56:45.105704 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.105672 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" event={"ID":"46765500-9cd2-4c86-a0cb-046a98b6ea69","Type":"ContainerStarted","Data":"75f743ab99297b054e0f54091390053dc1294aee60ec2b679f63910e8b20933d"} Nov 04 00:56:45.210841 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.210784 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" podStartSLOduration=3.510625971 podStartE2EDuration="14.210766503s" totalImagesPullingTime="10.700140532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:45.210277366 +0000 UTC m=+14.755306706" watchObservedRunningTime="2026-11-04 00:56:45.210766503 +0000 UTC m=+14.755795843" Nov 04 00:56:45.323280 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.323219 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" podStartSLOduration=4.009333452 podStartE2EDuration="14.323197438s" totalImagesPullingTime="10.313863986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:45.318247509 +0000 UTC m=+14.863276850" watchObservedRunningTime="2026-11-04 00:56:45.323197438 +0000 UTC m=+14.868226779" Nov 04 00:56:45.946814 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.946783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:45.946814 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.946817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:45.947056 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:45.946902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:45.947056 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:45.946977 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:45.947056 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:45.947016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:45.947176 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:45.947107 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:46.110329 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:46.110291 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="a22fd9e985453fa9c0012117fc8079c62ad3548661502680cd9a7dff60ec53e2" exitCode=0 Nov 04 00:56:46.111065 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:46.111037 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"a22fd9e985453fa9c0012117fc8079c62ad3548661502680cd9a7dff60ec53e2"} Nov 04 00:56:47.860018 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.859804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:47.860472 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.860052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:47.860472 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.859935 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:47.860472 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.860161 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:47.860472 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.860187 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret podName:c12d5c7d-b992-4093-8b4f-1bf058820213 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.860159239 +0000 UTC m=+33.405188559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret") pod "global-pull-secret-syncer-lkssf" (UID: "c12d5c7d-b992-4093-8b4f-1bf058820213") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:56:47.860472 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.860211 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.860200791 +0000 UTC m=+33.405230108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:56:47.946785 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.946758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:47.946961 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.946851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:47.946961 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.946857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:47.946961 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.946882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:47.947103 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.946970 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:47.947103 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.947049 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:47.960501 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:47.960482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:47.960647 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.960623 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:56:47.960647 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.960645 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:56:47.960757 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.960657 2579 projected.go:196] Error preparing data for projected volume kube-api-access-grr2k for pod openshift-network-diagnostics/network-check-target-56frc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:47.960757 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:47.960707 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k podName:94af8b5f-285c-47fe-9f67-f3b20f1d03c5 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.960690341 +0000 UTC m=+33.505719673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-grr2k" (UniqueName: "kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k") pod "network-check-target-56frc" (UID: "94af8b5f-285c-47fe-9f67-f3b20f1d03c5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:56:48.115512 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:48.115442 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="2674d555fa3d226606515341407c121732149e1a87de59f9100d28f925d0e699" exitCode=0 Nov 04 00:56:48.115512 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:48.115487 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"2674d555fa3d226606515341407c121732149e1a87de59f9100d28f925d0e699"} Nov 04 00:56:49.122182 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122009 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:56:49.122670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122505 2579 generic.go:356] "Generic (PLEG): container finished" podID="a7323316-cacf-4ca1-afb1-7b38b203dfd7" containerID="c5378304da37916f95824213c5267670e4094177974188621bc636fa339a1878" exitCode=1 Nov 04 00:56:49.122670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122540 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"326ff5962198e23d89622968610b9ec0e25d6225eab2516143301315abc382d8"} Nov 04 00:56:49.122670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122597 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"da9692fc3fca1488cb48b62e896535b6ec5802e2acfb31b294beaf0ad863ed16"} Nov 04 00:56:49.122670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122628 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"c13bcc0a96119f1b0a61c22c09976e7aefa48783e0274fc68bdbb79b21df0251"} Nov 04 00:56:49.122670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122656 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"9fafc15da9ce33ef9acee703e5ada7ebfb52ab83b4c667259214a1e06c0ae245"} Nov 04 00:56:49.122822 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122682 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerDied","Data":"c5378304da37916f95824213c5267670e4094177974188621bc636fa339a1878"} Nov 04 00:56:49.122822 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.122710 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"48f760028a7eb1d23bf298e67f7bf4c81033909c4b7eef6a32bd9607b03d6a78"} Nov 04 00:56:49.946050 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.946020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:49.946211 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.946020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:49.946211 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:49.946138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:49.946211 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:49.946020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:49.946211 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:49.946191 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:49.946372 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:49.946275 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:51.129104 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.129078 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:56:51.129573 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.129398 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"72910ecdd217c5793a279179a6991cb3223a68caf7d7a35b5268d7d255e633b9"} Nov 04 00:56:51.131677 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.131653 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="79b43ac394fee4a7d24fe3947e45f482dbbe99d2bd64e617115df28f0487f559" exitCode=0 Nov 04 00:56:51.131760 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.131690 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"79b43ac394fee4a7d24fe3947e45f482dbbe99d2bd64e617115df28f0487f559"} Nov 04 00:56:51.946357 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.946320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:51.946357 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.946343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:51.946571 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:51.946435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:51.946571 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:51.946448 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:51.946571 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:51.946507 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:51.946683 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:51.946579 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:52.135947 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:52.135921 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="0868f1304c9c3008a5537cb76841654e0a12f9dac999300f0566c53bdacd2370" exitCode=0 Nov 04 00:56:52.136332 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:52.135959 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerDied","Data":"0868f1304c9c3008a5537cb76841654e0a12f9dac999300f0566c53bdacd2370"} Nov 04 00:56:52.729235 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:52.729206 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:52.729462 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:52.729312 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:52.729812 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:52.729795 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:56:53.140908 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.140660 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g69bf" event={"ID":"e9fafd6c-f585-4d85-a956-b536ee09d521","Type":"ContainerStarted","Data":"f7393fc7d55a4cb12c36eee9c4e342addccd3c24947c254dde0a9aaae3d59b85"} Nov 04 00:56:53.143776 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.143755 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:56:53.144156 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.144131 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"c44601fa861cad2c4ea589d296d0977e7da203a2a26c8fed2d0b1bb53692d331"} Nov 04 00:56:53.144543 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.144527 2579 scope.go:122] "RemoveContainer" containerID="c5378304da37916f95824213c5267670e4094177974188621bc636fa339a1878" Nov 04 00:56:53.258060 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.258005 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g69bf" podStartSLOduration=10.314088399 podStartE2EDuration="22.257968426s" totalImagesPullingTime="11.943880027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:53.25766046 +0000 UTC m=+22.802689790" watchObservedRunningTime="2026-11-04 00:56:53.257968426 +0000 UTC m=+22.802997765" Nov 04 00:56:53.946724 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.946690 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:53.946724 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.946708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:53.946928 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:53.946797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:53.946928 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:53.946876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:53.947005 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:53.946931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:53.947035 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:53.947021 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:54.152010 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.151968 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:56:54.153825 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.153795 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:54.154406 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.154362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" event={"ID":"a7323316-cacf-4ca1-afb1-7b38b203dfd7","Type":"ContainerStarted","Data":"ced60ca7b1344c551c875784aa4f98bcf3776e4ed82cbb540ed2fcf99b4b1093"} Nov 04 00:56:54.154647 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.154631 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:54.155557 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.155540 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:54.174113 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.174086 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:54.174609 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.174592 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:54.318145 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:54.318100 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" podStartSLOduration=8.276622084 podStartE2EDuration="23.318084221s" totalImagesPullingTime="15.041462137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:56:54.317378907 +0000 UTC m=+23.862408246" watchObservedRunningTime="2026-11-04 00:56:54.318084221 +0000 UTC m=+23.863113562" Nov 04 00:56:55.155894 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.155866 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:56:55.361331 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.361297 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:56:55.508931 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.508857 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56frc"] Nov 04 00:56:55.509088 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.508973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:55.509160 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:55.509092 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:55.511461 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.511427 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkssf"] Nov 04 00:56:55.511578 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.511533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:55.511638 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:55.511622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:55.512047 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.512029 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-slvp5"] Nov 04 00:56:55.512131 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:55.512118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:55.512217 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:55.512201 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:56.946695 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:56.946663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:56.947250 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:56.946671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:56.947250 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:56.946805 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:56.947250 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:56.946671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:56.947250 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:56.946851 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:56:56.947250 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:56.946921 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:57.169854 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:57.169818 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" podUID="a7323316-cacf-4ca1-afb1-7b38b203dfd7" containerName="ovnkube-controller" probeResult="failure" output="" Nov 04 00:56:58.946314 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:58.946283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:56:58.946314 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:58.946300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:56:58.946949 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:56:58.946302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:56:58.946949 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:58.946404 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:56:58.946949 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:58.946451 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:56:58.946949 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:56:58.946517 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:57:00.946898 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:00.946732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:00.946898 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:00.946885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:00.947357 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:00.946822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:00.947357 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:00.946971 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-56frc" podUID="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" Nov 04 00:57:00.947357 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:00.947055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lkssf" podUID="c12d5c7d-b992-4093-8b4f-1bf058820213" Nov 04 00:57:00.947357 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:00.947122 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-slvp5" podUID="469899a6-621d-4421-9879-eba5b3c63f1e" Nov 04 00:57:01.172064 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.172035 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeReady" Nov 04 00:57:01.172209 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.172168 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 00:57:01.313304 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.313277 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-drlm7"] Nov 04 00:57:01.316674 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.316656 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lmxws"] Nov 04 00:57:01.316803 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.316785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.320085 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.320068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.335972 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.335954 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmxws"] Nov 04 00:57:01.338739 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.338715 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-drlm7"] Nov 04 00:57:01.404306 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404276 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:57:01.404306 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404278 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.404306 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404304 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8chh7\"" Nov 04 00:57:01.404516 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404314 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:57:01.404516 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404277 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:57:01.404516 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404352 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g2z88\"" Nov 04 00:57:01.404516 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.404420 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.451797 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.451765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.451944 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.451822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.451944 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.451845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.451944 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.451914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdc7\" (UniqueName: \"kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.452121 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.452045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq48\" (UniqueName: \"kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.452121 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.452068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.553315 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq48\" (UniqueName: \"kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.553444 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.553530 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.553590 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.553590 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.553670 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.553605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdc7\" (UniqueName: \"kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.553730 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:01.553713 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:01.553781 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:01.553770 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.053753932 +0000 UTC m=+31.598783269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:01.554080 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.554057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.554288 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.554271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.557755 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.557733 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.609007 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.608931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq48\" (UniqueName: \"kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:01.609290 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.609271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdc7\" (UniqueName: \"kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:01.635183 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:01.635162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:02.009415 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.009336 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lmxws"] Nov 04 00:57:02.013149 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:57:02.013122 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51861d34_b375_4586_bc20_472ed805e809.slice/crio-55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a WatchSource:0}: Error finding container 55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a: Status 404 returned error can't find the container with id 55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a Nov 04 00:57:02.057355 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.057335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:02.057473 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.057463 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:02.057539 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.057529 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.057511055 +0000 UTC m=+32.602540395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:02.170425 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.170391 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmxws" event={"ID":"51861d34-b375-4586-bc20-472ed805e809","Type":"ContainerStarted","Data":"55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a"} Nov 04 00:57:02.259706 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.259619 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nsgcc"] Nov 04 00:57:02.264066 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.264051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.317000 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.316942 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:02.317150 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.317018 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:57:02.317150 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.317048 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:57:02.317150 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.317129 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7knl\"" Nov 04 00:57:02.317150 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.317144 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:02.318199 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.318179 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nsgcc"] Nov 04 00:57:02.358469 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.358444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.358579 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.358488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.358579 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.358520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.358678 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.358651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.358678 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.358670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z9gf\" (UniqueName: \"kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459329 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459496 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459496 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459570 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459570 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459628 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.459577 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:02.459628 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9gf\" (UniqueName: \"kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459688 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.459663 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.959614177 +0000 UTC m=+32.504643521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : configmap "kube-rbac-proxy" not found Nov 04 00:57:02.459688 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.459674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.459753 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.459695 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.459784 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.459753 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.959739904 +0000 UTC m=+32.504769252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.477393 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.477356 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:02.480599 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.480579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.515362 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.515289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9gf\" (UniqueName: \"kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.529522 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.529495 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:57:02.529522 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.529496 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lbk4\"" Nov 04 00:57:02.560658 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.560625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.560827 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.560707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.560827 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.560774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.560927 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.560892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.603894 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.603860 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:57:02.604578 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.604558 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:02.662139 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.662320 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.662320 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.662320 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.662644 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.662845 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.662824 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.663075 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.663056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.716326 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.716302 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.791523 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.791440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:02.949879 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.949847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:02.950054 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.949921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:02.950054 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.950044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:02.965212 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.965193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.965316 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.965238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:02.965316 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.965300 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:02.965383 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.965329 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.965383 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.965348 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.965335632 +0000 UTC m=+33.510364949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : configmap "kube-rbac-proxy" not found Nov 04 00:57:02.965383 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:02.965376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:03.965366095 +0000 UTC m=+33.510395449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:02.999803 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.999777 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:57:02.999905 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.999868 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:02.999905 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.999879 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:57:02.999905 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.999903 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-47hrz\"" Nov 04 00:57:03.000110 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:02.999942 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fh9wk\"" Nov 04 00:57:03.000110 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.000035 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xq5nc\"" Nov 04 00:57:03.000203 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.000136 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:03.065463 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.065435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:03.065764 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.065565 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:03.065764 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.065625 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:05.065607938 +0000 UTC m=+34.610637272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:03.152284 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.152246 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:03.156300 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:57:03.156260 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846abc03_803a_4e83_839c_0cab2a41dc7b.slice/crio-32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563 WatchSource:0}: Error finding container 32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563: Status 404 returned error can't find the container with id 32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563 Nov 04 00:57:03.174502 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.174477 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" event={"ID":"846abc03-803a-4e83-839c-0cab2a41dc7b","Type":"ContainerStarted","Data":"32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563"} Nov 04 00:57:03.870806 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.870769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:03.870958 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.870828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:03.870958 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.870934 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 00:57:03.871075 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.871003 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs podName:469899a6-621d-4421-9879-eba5b3c63f1e nodeName:}" failed. No retries permitted until 2026-11-04 00:57:35.870974518 +0000 UTC m=+65.416003835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs") pod "network-metrics-daemon-slvp5" (UID: "469899a6-621d-4421-9879-eba5b3c63f1e") : secret "metrics-daemon-secret" not found Nov 04 00:57:03.873079 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.873059 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:03.875007 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.874993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:03.971591 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.971559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:03.971756 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.971611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:03.971756 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.971655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:03.971896 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.971757 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:03.971896 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.971770 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:03.971896 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.971829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:05.971810616 +0000 UTC m=+35.516839957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : configmap "kube-rbac-proxy" not found Nov 04 00:57:03.971896 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:03.971846 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:05.971839985 +0000 UTC m=+35.516869302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:03.974327 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:03.974304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:04.167530 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.167465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:04.182143 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.181579 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-87hnt"] Nov 04 00:57:04.185323 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.185305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.234785 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.234757 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lkssf"] Nov 04 00:57:04.237435 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.237282 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 00:57:04.237435 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.237346 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 00:57:04.237435 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.237282 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 00:57:04.237435 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.237393 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wmbbs\"" Nov 04 00:57:04.237435 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.237416 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 00:57:04.238332 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.238267 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-87hnt"] Nov 04 00:57:04.245251 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:57:04.245226 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12d5c7d_b992_4093_8b4f_1bf058820213.slice/crio-320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8 WatchSource:0}: Error finding container 320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8: Status 404 returned error can't find the container with id 320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8 Nov 04 00:57:04.273951 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.273927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.274117 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.273996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqfvp\" (UniqueName: \"kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.274117 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.274065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.375283 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.375251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqfvp\" (UniqueName: \"kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.375451 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.375332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.375451 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.375372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.375999 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.375946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.378226 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.378208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.428567 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.428505 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqfvp\" (UniqueName: \"kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.498533 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.498503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:04.532456 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.532430 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-56frc"] Nov 04 00:57:04.539065 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:57:04.539027 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94af8b5f_285c_47fe_9f67_f3b20f1d03c5.slice/crio-15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7 WatchSource:0}: Error finding container 15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7: Status 404 returned error can't find the container with id 15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7 Nov 04 00:57:04.868000 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:04.867955 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-87hnt"] Nov 04 00:57:04.871307 ip-10-0-129-114 kubenswrapper[2579]: W1104 00:57:04.871280 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39c1521_1467_4a9c_a993_ef855ebe4a0a.slice/crio-dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942 WatchSource:0}: Error finding container dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942: Status 404 returned error can't find the container with id dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942 Nov 04 00:57:05.083210 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.083172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:05.083352 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:05.083329 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:05.083423 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:05.083408 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.083387695 +0000 UTC m=+38.628417040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:05.181719 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.181673 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56frc" event={"ID":"94af8b5f-285c-47fe-9f67-f3b20f1d03c5","Type":"ContainerStarted","Data":"15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7"} Nov 04 00:57:05.183104 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.183075 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lkssf" event={"ID":"c12d5c7d-b992-4093-8b4f-1bf058820213","Type":"ContainerStarted","Data":"320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8"} Nov 04 00:57:05.184476 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.184449 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" event={"ID":"c39c1521-1467-4a9c-a993-ef855ebe4a0a","Type":"ContainerStarted","Data":"dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942"} Nov 04 00:57:05.990776 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.990734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:05.990937 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.990811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:05.991060 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:05.990942 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:05.991060 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:05.991020 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:09.991006054 +0000 UTC m=+39.536035384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:05.991267 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:05.991243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:07.198708 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:07.198649 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" event={"ID":"846abc03-803a-4e83-839c-0cab2a41dc7b","Type":"ContainerStarted","Data":"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0"} Nov 04 00:57:07.309158 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:07.309088 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" podStartSLOduration=2.395728555 podStartE2EDuration="5.309071875s" totalImagesPullingTime="2.91334332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:07.308895982 +0000 UTC m=+36.853925321" watchObservedRunningTime="2026-11-04 00:57:07.309071875 +0000 UTC m=+36.854101214" Nov 04 00:57:08.203885 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:08.203851 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lmxws" event={"ID":"51861d34-b375-4586-bc20-472ed805e809","Type":"ContainerStarted","Data":"d7661a69486a966a315e3f7216dcbaa433c4449b4889fa9c58f059961a44ab6a"} Nov 04 00:57:08.322354 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:08.322291 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lmxws" podStartSLOduration=1.374455481 podStartE2EDuration="7.32227062s" totalImagesPullingTime="5.947815139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:08.32050433 +0000 UTC m=+37.865533668" watchObservedRunningTime="2026-11-04 00:57:08.32227062 +0000 UTC m=+37.867299959" Nov 04 00:57:09.118003 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:09.117951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:09.118184 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:09.118104 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:09.118184 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:09.118169 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:17.118151244 +0000 UTC m=+46.663180574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:09.874674 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:09.874642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7g22g_692caf64-a543-4631-83a9-ce22eb1ec34a/dns-node-resolver/0.log" Nov 04 00:57:10.026409 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.026341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:10.026593 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:10.026496 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:10.026593 ip-10-0-129-114 kubenswrapper[2579]: E1104 00:57:10.026560 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls podName:bba37d6d-78c6-4e30-a76a-16cf6c61f424 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:18.026545333 +0000 UTC m=+47.571574669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nsgcc" (UID: "bba37d6d-78c6-4e30-a76a-16cf6c61f424") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:10.211451 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.211408 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-56frc" event={"ID":"94af8b5f-285c-47fe-9f67-f3b20f1d03c5","Type":"ContainerStarted","Data":"ec4bbd7cc8c6ff22de509759acc7d3aaa4218f493e637caecfbd96977688b035"} Nov 04 00:57:10.211625 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.211500 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:10.213234 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.213205 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" event={"ID":"c39c1521-1467-4a9c-a993-ef855ebe4a0a","Type":"ContainerStarted","Data":"b3cee83b9c641e695dac71af5cef520d873f2eea2ddf5762a3f477085920ac80"} Nov 04 00:57:10.316309 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.315939 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-56frc" podStartSLOduration=34.187985343 podStartE2EDuration="39.315922413s" totalImagesPullingTime="5.12793707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:10.315136379 +0000 UTC m=+39.860165720" watchObservedRunningTime="2026-11-04 00:57:10.315922413 +0000 UTC m=+39.860951757" Nov 04 00:57:10.422191 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:10.422134 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" podStartSLOduration=1.865139831 podStartE2EDuration="6.42211318s" totalImagesPullingTime="4.556973349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:10.421216689 +0000 UTC m=+39.966246039" watchObservedRunningTime="2026-11-04 00:57:10.42211318 +0000 UTC m=+39.967142518" Nov 04 00:57:11.074651 ip-10-0-129-114 kubenswrapper[2579]: I1104 00:57:11.074617 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5scf_33d17845-e07f-43a7-b2bf-c73d741f180f/node-ca/0.log" Nov 04 00:57:11.184957 ip-10-0-129-114 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 00:57:11.195517 ip-10-0-129-114 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 00:57:11.195735 ip-10-0-129-114 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 00:57:11.195790 ip-10-0-129-114 systemd[1]: kubelet.service: Consumed 2.427s CPU time. Nov 04 00:57:11.212773 ip-10-0-129-114 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:57:11.268766 ip-10-0-129-114 kubenswrapper[4486]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:11.268766 ip-10-0-129-114 kubenswrapper[4486]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:57:11.268766 ip-10-0-129-114 kubenswrapper[4486]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:11.268766 ip-10-0-129-114 kubenswrapper[4486]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:11.270677 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270612 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.270677 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270678 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270683 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270686 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270689 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270692 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270695 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270697 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270700 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270703 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270706 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270708 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270711 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270714 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270717 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270720 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270722 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270725 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270728 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270731 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.270759 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270734 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270737 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270740 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270742 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270745 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270747 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270750 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270753 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270756 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270759 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270761 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270764 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270767 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270770 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270774 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270776 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270779 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270781 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270784 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270786 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.271252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270789 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270791 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270794 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270797 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270800 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270803 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270805 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270808 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270810 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270813 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270816 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270819 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270821 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270824 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270827 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270830 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270832 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270835 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270837 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270840 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.271749 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270842 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270845 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270848 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270851 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270853 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270856 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270858 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270861 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270864 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270866 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270869 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270871 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270874 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270876 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270879 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270881 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270885 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270887 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270890 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270893 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.272273 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270895 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270898 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270901 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270904 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270907 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270909 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270912 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270915 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270917 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270922 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270926 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270929 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270932 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270935 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270938 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270940 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270943 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270946 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.272760 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270950 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270954 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270958 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270961 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270963 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270966 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270968 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.270971 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271714 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271722 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271726 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271731 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271735 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271738 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271740 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271743 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271746 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271749 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271752 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.273245 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271754 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271757 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271759 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271762 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271765 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271768 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271770 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271773 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271776 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271779 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271781 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271784 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271786 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271789 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271791 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271794 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271796 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271799 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271801 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.273722 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271804 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271806 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271809 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271812 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271815 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271817 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271826 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271828 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271831 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271833 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271836 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271838 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271841 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271843 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271846 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271849 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271852 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271855 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271858 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271860 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.274258 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271863 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271866 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271868 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271871 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271873 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271876 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271879 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271882 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271884 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271887 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271889 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271892 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271894 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271897 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271914 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271919 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271922 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271925 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271928 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271936 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.274763 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271939 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271941 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271944 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271946 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271949 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271953 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271956 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271960 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271963 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271966 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271969 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271972 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271975 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271977 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.271996 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272000 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272003 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272006 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272015 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.275264 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272018 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272020 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272023 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272026 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272029 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272031 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272034 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272037 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272040 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272042 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272045 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272048 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272051 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272059 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272062 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272065 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272069 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272526 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272531 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272534 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.275730 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272537 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272540 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272542 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272545 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272548 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272550 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272554 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272557 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272560 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272563 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272566 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272569 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272571 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272574 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272576 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272579 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272582 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272584 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.276293 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272591 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272595 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272598 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272601 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272603 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272606 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272608 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272611 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272615 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272617 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272620 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272623 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272625 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272628 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272630 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272633 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272635 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272638 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272640 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272643 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.277143 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272645 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272648 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272650 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272653 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272656 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272658 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272661 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272663 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272666 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272669 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272672 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272675 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272682 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272685 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272687 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272690 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272693 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272695 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272698 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272700 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.277846 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272703 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272706 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272709 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272711 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272714 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272716 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272719 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272721 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272724 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272726 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272729 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272732 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272734 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272737 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272739 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272742 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272744 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272747 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272750 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272752 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.278373 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272755 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272757 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272760 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272762 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272768 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272770 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272773 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272776 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272778 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272781 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272783 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272786 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272788 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272791 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272794 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272797 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272799 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272802 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272804 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272807 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.278860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272809 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272812 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272814 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272817 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.272821 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272908 4486 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272915 4486 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272922 4486 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272926 4486 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272930 4486 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272934 4486 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272938 4486 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272942 4486 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272946 4486 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272949 4486 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272952 4486 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272958 4486 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272961 4486 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272964 4486 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272967 4486 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272970 4486 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272973 4486 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272976 4486 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:57:11.279366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.272996 4486 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273000 4486 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273003 4486 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273008 4486 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273012 4486 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273015 4486 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273018 4486 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273022 4486 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273025 4486 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273028 4486 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273031 4486 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273034 4486 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273037 4486 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273040 4486 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273045 4486 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273048 4486 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273051 4486 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273054 4486 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273057 4486 flags.go:64] FLAG: --enable-server="true" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273060 4486 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273064 4486 flags.go:64] FLAG: --event-burst="100" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273067 4486 flags.go:64] FLAG: --event-qps="50" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273070 4486 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273074 4486 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:57:11.280090 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273077 4486 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273084 4486 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273088 4486 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273091 4486 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273094 4486 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273097 4486 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273100 4486 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273103 4486 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273106 4486 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273109 4486 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273112 4486 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273115 4486 flags.go:64] FLAG: --feature-gates="" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273119 4486 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273123 4486 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273126 4486 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273129 4486 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273132 4486 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273135 4486 flags.go:64] FLAG: --help="false" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273138 4486 flags.go:64] FLAG: --hostname-override="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273141 4486 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273144 4486 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273147 4486 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273151 4486 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:57:11.280716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273154 4486 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273157 4486 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273160 4486 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273163 4486 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273166 4486 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273170 4486 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273173 4486 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273176 4486 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273179 4486 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273182 4486 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273188 4486 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273191 4486 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273194 4486 flags.go:64] FLAG: --lock-file="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273197 4486 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273200 4486 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273203 4486 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273208 4486 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273211 4486 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273214 4486 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273218 4486 flags.go:64] FLAG: --logging-format="text" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273221 4486 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273224 4486 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273227 4486 flags.go:64] FLAG: --manifest-url="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273230 4486 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273235 4486 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:57:11.281311 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273238 4486 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273242 4486 flags.go:64] FLAG: --max-pods="110" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273245 4486 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273248 4486 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273251 4486 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273254 4486 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273258 4486 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273261 4486 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273264 4486 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273270 4486 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273273 4486 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273276 4486 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273279 4486 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273282 4486 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273285 4486 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273288 4486 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273292 4486 flags.go:64] FLAG: --port="10250" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273297 4486 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273300 4486 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0075e5eec87743520" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273303 4486 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273306 4486 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273309 4486 flags.go:64] FLAG: --register-node="true" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273312 4486 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273316 4486 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273319 4486 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273322 4486 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:57:11.281945 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273324 4486 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273328 4486 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273331 4486 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273334 4486 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273338 4486 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273341 4486 flags.go:64] FLAG: --runonce="false" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273344 4486 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273347 4486 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273350 4486 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273353 4486 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273356 4486 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273359 4486 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273362 4486 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273365 4486 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273368 4486 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273371 4486 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273373 4486 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273377 4486 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273380 4486 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273383 4486 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273385 4486 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273391 4486 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273394 4486 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273398 4486 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273402 4486 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:57:11.282611 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273405 4486 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273408 4486 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273411 4486 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273414 4486 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273417 4486 flags.go:64] FLAG: --v="2" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273421 4486 flags.go:64] FLAG: --version="false" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273425 4486 flags.go:64] FLAG: --vmodule="" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273429 4486 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273432 4486 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273555 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273559 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273562 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273565 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273568 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273571 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273574 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273576 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273579 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273582 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273584 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273587 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.283388 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273591 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273595 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273598 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273601 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273604 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273606 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273610 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273613 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273616 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273621 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273624 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273627 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273629 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273632 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273635 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273638 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273640 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273644 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273646 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273649 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.284110 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273651 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273654 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273657 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273659 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273663 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273666 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273668 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273671 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273674 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273676 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273679 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273681 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273684 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273686 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273689 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273691 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273694 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273696 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273699 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.284713 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273702 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273704 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273709 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273712 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273714 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273716 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273719 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273722 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273725 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273728 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273730 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273733 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273736 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273738 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273741 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273744 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273746 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273749 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273752 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273755 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.285259 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273757 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273760 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273763 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273765 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273768 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273770 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273773 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273775 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273777 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273780 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273783 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273785 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273788 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273790 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273795 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273798 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273801 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273803 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273806 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.285743 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273808 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273811 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273814 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273816 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273819 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273821 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273824 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273827 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273829 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273832 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273834 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273837 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273840 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273842 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273845 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273847 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.286481 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.273872 4486 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273911 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273915 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273918 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273921 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273924 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273926 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273929 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273931 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273934 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273937 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273941 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273945 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273948 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273950 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273953 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273956 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273958 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273960 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.286961 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273963 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273966 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273969 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273971 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273974 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273977 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273993 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273997 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.273999 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274002 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274005 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274008 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274011 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274013 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274016 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274019 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274021 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274024 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274026 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274029 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.287452 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274031 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274034 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274036 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274039 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274043 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274047 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274050 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274053 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274056 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274058 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274061 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274064 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274066 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274069 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274072 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274075 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274077 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274080 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274082 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.287934 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274086 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274090 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274093 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274096 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274100 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274102 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274106 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274110 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274113 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274115 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274119 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274121 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274124 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274126 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274130 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274133 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274135 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274139 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274143 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274146 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.288412 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274149 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274152 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274154 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274157 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274160 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274163 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274165 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274168 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274171 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274173 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274176 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274178 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274181 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274183 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274186 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274188 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274191 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274194 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274197 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.288935 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274199 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274202 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274205 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274207 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274210 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274212 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274215 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274218 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274220 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.274223 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.278928 4486 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.278944 4486 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.278964 4486 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.278968 4486 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279075 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279081 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279086 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279091 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279094 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.289416 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279097 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279100 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279103 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279107 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279109 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279112 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279115 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279118 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279120 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279123 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279126 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279129 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279131 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279134 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279137 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279139 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279142 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279145 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279147 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279150 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.289885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279153 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279155 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279158 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279161 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279163 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279166 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279169 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279171 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279174 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279176 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279179 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279181 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279184 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279186 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279189 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279192 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279194 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279197 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279200 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279202 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.290424 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279205 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279208 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279210 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279213 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279216 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279220 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279224 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279227 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279230 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279233 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279236 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279238 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279241 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279243 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279246 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279248 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279251 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279254 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279256 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.290906 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279259 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279262 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279265 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279267 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279270 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279272 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279275 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279277 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279280 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279282 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279285 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279288 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279290 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279293 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279295 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279299 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279302 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279304 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279307 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279309 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.291383 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279312 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279315 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279317 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279320 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279323 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279326 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279328 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279331 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279333 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279336 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279338 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279341 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279343 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279346 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279348 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279351 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279353 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279356 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279358 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279361 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.291860 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279363 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279366 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.279393 4486 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279543 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279547 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279550 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279553 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279556 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279560 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279563 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279565 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279568 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279571 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279573 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279576 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279578 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.292354 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279581 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279584 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279587 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279589 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279592 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279594 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279597 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279600 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279602 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279605 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279607 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279610 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279612 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279615 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279617 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279620 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279623 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279626 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279628 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.292734 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279631 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279634 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279636 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279639 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279643 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279647 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279650 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279653 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279656 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279658 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279661 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279663 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279666 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279668 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279671 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279674 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279676 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279679 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279682 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.293370 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279684 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279687 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279689 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279692 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279694 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279697 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279699 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279702 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279706 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279709 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279712 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279714 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279717 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279719 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279722 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279724 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279727 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279731 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279734 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.294156 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279737 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279740 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279742 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279745 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279747 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279750 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279753 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279756 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279758 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279760 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279763 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279766 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279768 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279771 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279773 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279776 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279778 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279781 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279783 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279786 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.294716 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279788 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279791 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279794 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279796 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279799 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279803 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279805 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279808 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279811 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279813 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279817 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279820 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279823 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279825 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279828 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279831 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.295252 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.279854 4486 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279884 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279889 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279892 4486 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279894 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279897 4486 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279900 4486 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279902 4486 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279905 4486 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279908 4486 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279910 4486 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279913 4486 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279916 4486 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279918 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279921 4486 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279923 4486 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279926 4486 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279928 4486 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279931 4486 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279934 4486 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:11.295648 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279936 4486 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279939 4486 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279943 4486 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279946 4486 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279949 4486 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279951 4486 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279956 4486 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279958 4486 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279961 4486 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279964 4486 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279966 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279970 4486 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279972 4486 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279975 4486 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279978 4486 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.279996 4486 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280000 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280005 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280015 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280018 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:11.296146 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280021 4486 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280024 4486 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280026 4486 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280029 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280032 4486 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280034 4486 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280037 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280040 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280043 4486 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280046 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280048 4486 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280051 4486 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280055 4486 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280060 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280066 4486 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280069 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280072 4486 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280075 4486 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280079 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:11.296753 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280081 4486 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280084 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280087 4486 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280089 4486 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280092 4486 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280096 4486 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280099 4486 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280102 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280105 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280108 4486 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280110 4486 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280113 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280115 4486 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280118 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280120 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280123 4486 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280127 4486 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280131 4486 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280134 4486 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:11.297240 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280137 4486 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280140 4486 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280142 4486 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280145 4486 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280147 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280150 4486 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280153 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280155 4486 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280159 4486 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280162 4486 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280165 4486 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280168 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280171 4486 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280174 4486 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280176 4486 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280179 4486 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280181 4486 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280184 4486 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280186 4486 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280189 4486 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:11.297790 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280197 4486 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280201 4486 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280204 4486 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280207 4486 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280209 4486 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280212 4486 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280214 4486 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280217 4486 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:11.280220 4486 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.280274 4486 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281208 4486 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281255 4486 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281424 4486 server.go:1014] "Starting client certificate rotation" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281566 4486 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281621 4486 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:30 +0000 UTC" deadline="2028-07-10 04:51:05.04945551 +0000 UTC" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281656 4486 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14739h53m53.767803613s" Nov 04 00:57:11.298408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.281859 4486 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:11.298853 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.282923 4486 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:11.298853 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.284336 4486 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:57:11.298853 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.285919 4486 log.go:25] "Validated CRI v1 image API" Nov 04 00:57:11.298853 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.286718 4486 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:57:11.298853 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.288171 4486 fs.go:135] Filesystem UUIDs: map[102c7c3d-248f-4798-acbf-0e6ed155630a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc83f719-d469-4699-bfca-9d4d0de43477:/dev/nvme0n1p3] Nov 04 00:57:11.299043 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.288186 4486 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/461614a38d17871fba18f370edd54bc112b85cc54e338e1cfafdaf73b9104750/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/461614a38d17871fba18f370edd54bc112b85cc54e338e1cfafdaf73b9104750/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10a00aa9-5c78-48fc-8baf-40efe65c9db6/volumes/kubernetes.io~projected/kube-api-access-8pq48:{mountpoint:/var/lib/kubelet/pods/10a00aa9-5c78-48fc-8baf-40efe65c9db6/volumes/kubernetes.io~projected/kube-api-access-8pq48 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d17845-e07f-43a7-b2bf-c73d741f180f/volumes/kubernetes.io~projected/kube-api-access-crdtm:{mountpoint:/var/lib/kubelet/pods/33d17845-e07f-43a7-b2bf-c73d741f180f/volumes/kubernetes.io~projected/kube-api-access-crdtm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46765500-9cd2-4c86-a0cb-046a98b6ea69/volumes/kubernetes.io~projected/kube-api-access-xlntm:{mountpoint:/var/lib/kubelet/pods/46765500-9cd2-4c86-a0cb-046a98b6ea69/volumes/kubernetes.io~projected/kube-api-access-xlntm major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/469899a6-621d-4421-9879-eba5b3c63f1e/volumes/kubernetes.io~projected/kube-api-access-xrp25:{mountpoint:/var/lib/kubelet/pods/469899a6-621d-4421-9879-eba5b3c63f1e/volumes/kubernetes.io~projected/kube-api-access-xrp25 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~projected/kube-api-access-mbdc7:{mountpoint:/var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~projected/kube-api-access-mbdc7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/692caf64-a543-4631-83a9-ce22eb1ec34a/volumes/kubernetes.io~projected/kube-api-access-bfcs4:{mountpoint:/var/lib/kubelet/pods/692caf64-a543-4631-83a9-ce22eb1ec34a/volumes/kubernetes.io~projected/kube-api-access-bfcs4 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75d094ba-dfa9-48bc-8ae7-028003bae8a8/volumes/kubernetes.io~projected/kube-api-access-r2tsl:{mountpoint:/var/lib/kubelet/pods/75d094ba-dfa9-48bc-8ae7-028003bae8a8/volumes/kubernetes.io~projected/kube-api-access-r2tsl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/846abc03-803a-4e83-839c-0cab2a41dc7b/volumes/kubernetes.io~projected/kube-api-access-jmp28:{mountpoint:/var/lib/kubelet/pods/846abc03-803a-4e83-839c-0cab2a41dc7b/volumes/kubernetes.io~projected/kube-api-access-jmp28 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/858fcc1f-21e9-4e03-b22e-115a8377a8fc/volumes/kubernetes.io~projected/kube-api-access-bpm9d:{mountpoint:/var/lib/kubelet/pods/858fcc1f-21e9-4e03-b22e-115a8377a8fc/volumes/kubernetes.io~projected/kube-api-access-bpm9d major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94af8b5f-285c-47fe-9f67-f3b20f1d03c5/volumes/kubernetes.io~projected/kube-api-access-grr2k:{mountpoint:/var/lib/kubelet/pods/94af8b5f-285c-47fe-9f67-f3b20f1d03c5/volumes/kubernetes.io~projected/kube-api-access-grr2k major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~projected/kube-api-access-cn8f6:{mountpoint:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~projected/kube-api-access-cn8f6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bba37d6d-78c6-4e30-a76a-16cf6c61f424/volumes/kubernetes.io~projected/kube-api-access-9z9gf:{mountpoint:/var/lib/kubelet/pods/bba37d6d-78c6-4e30-a76a-16cf6c61f424/volumes/kubernetes.io~projected/kube-api-access-9z9gf major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c12d5c7d-b992-4093-8b4f-1bf058820213/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c12d5c7d-b992-4093-8b4f-1bf058820213/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~projected/kube-api-access-hqfvp:{mountpoint:/var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~projected/kube-api-access-hqfvp major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~secret/signing-key major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~projected/kube-api-access-wfxcw:{mountpoint:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~projected/kube-api-access-wfxcw major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9fafd6c-f585-4d85-a956-b536ee09d521/volumes/kubernetes.io~projected/kube-api-access-bg2xd:{mountpoint:/var/lib/kubelet/pods/e9fafd6c-f585-4d85-a956-b536ee09d521/volumes/kubernetes.io~projected/kube-api-access-bg2xd major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9554504-e458-45eb-ac2f-fb8bd160802e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f9554504-e458-45eb-ac2f-fb8bd160802e/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7536b88312eaa5c3196c72769b35619df2d28e4efdc5c7b1caf22671eae0eb6f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/42073736c5b71ad73f259d4aa69a2ecd4561aa073503ff19ca570ad209619d65/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/897efadd17308966f311ec567d64025da987c9f4cbaa972242803839b458508c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2e6b6475edfaa2e3051799e655bdafcb3d6f7cc812cdc8da5f834526df8db23b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5d88f85fec6bd7b3267a25347d9fd4b8bae787ddd476a4f9b419b2f6e1c50b87/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/d04396c3ad36d5928f6bb5457a961ee33441e4bc737b946b2ff5e10969496ab8/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/96bf8607b5a06d22e27950603eb477c5e5279d8bb3d53c27b99a91c013d1aa8b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d99e8f8658d98f33c7f8bc3232ad60db0a8ee75ce82c12d8bfdf10ce4547283c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5ad216124806b8090bc534854d6e6f2fca6af0609c1ac7f09d6ea3128453036a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/e71a534d312f6ed355eda40b9e848e371d5a8b7ad44d0734c7fb87cb47749b34/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6670767b38522347ef69fbec10fb97b1f260ae6cb1aa93ec7e123a846220aee0/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3ad87877670e399587545775aef6054eec90bca1745b839a8ca0d687d7de7d32/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/49400fadb1e8650a1850bbdd1ccec38551111f604cb8aa6f36d159b77d30d195/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0529f182e5e004b839feacfd6f339dd2c6588bba0169ac76ec6fba7ddb97ed63/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/15403eb93be107e6c7c350e743073625a2234e16de5f928e04de3de853fdffd4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/12ceae2fa590259f25e4445b008053e8dd85ef552f6ccc7920713cc827c9cedf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/5f6f16a52f58d62820dcc8a176a1eb99bb8031cb86a087f2daac0adb4c103013/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/152b630039df24bac61bc93852ce8a1b72b4daab13b40793f1f4aab0053e9530/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cf4e293dae9630a475112be3273cfc2660dca83010d35238122601422bc0b3e9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/48bbddb9e9ed8557442b6f3204dfea7758d54c6eb5cd0d6078bcebf2745a0c34/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4c82ee6ea6bf8889ddd3f34b9024a598390336dc06ccb1b9ba54701c142925b3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/3f99d4b6ad8c4b7acbdc31d59637d681fed5b1461f49761ede9e7650692c66fd/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/4fac539fc42c3205cdec87a5ce32c9615d245ca3868633ccf08c5e5c85a8717d/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/33e49e262dd3329dee3a5eadb8b7bbd4fcfa3af6e0010c17fe54e872de81d43d/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/b3b555b0f543d151a69ad9f6c9c8dffa745e2728715e859de5bf346093e6fbf6/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/2b7842ab5601df4f4966e7a75d0af1e942c11aefcbacdd2a7c9aa46fb378981d/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/27045deac0cbc0c2fe16654416f5d8575495dd90238eb98d58f4371496c9ecfa/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/a6c88b239c5f5e090a11ed73a55e7e98fbdf0d08a666ca3b403c9410fd9ceb00/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/199ee8f18c1bd56d01909f1048ebf3285047a2bc0486f21878c7127d2d75bcfc/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/0274d9ba10e26221b9d8c67d8bde299eb305ea6b43fc6f63a6077e476a8bcc5c/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/58915fa230e07477686be5b39005cdb3ae97b1a52ba831549d121b185794dc26/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/d889610a9a87fbab9a7b15722f0a9c277b5cb7dfc16558add4f2a1eaa8bdef11/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/3e6fd072f6062a74f3552ce70a3fa6545baa1704d36c53285df43d872491ce68/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f7a0f300804223c87c252f1658d13e5ede0bcb9914a545eb51ff9d7bf9ca0309/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8afe1f314fab5104d03379de494bd22be92ff541388e01a0f48719fa47be1b89/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e8dd1c3078a03b02cd68bdcd0d8072c2e9c1e1757f5b8c70c426d53eeb0175b2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/09b770501f14d591ad0b0320e0cfbd06c7dd4b7994d9707e93b262a9ac955818/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ad50cf04ecf18d77c338324d5f19acc0d53fd4dd3d8e3b95f4a7cf37493966e9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/0dca555ba9bfca885f2cfab5af4fd5836240f5e1420c7efbf390525b3dc28e42/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/da2a56d26fc10416049f240ce55478e993c1d7b9cd77bcf39ad332a6109e7c2c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f7222cb9fd4619412a16ccb1811d7cbc2bf9174f8ea64db98bcbeedb55634dbb/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296058 4486 manager.go:217] Machine: {Timestamp:2026-11-04 00:57:11.295580462 +0000 UTC m=+0.054242614 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3079105 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dcbb1326988890659d34018b66fbe SystemUUID:ec2dcbb1-3269-8889-0659-d34018b66fbe BootID:9cb3c6d0-89ba-4a68-9592-92117f221526 Filesystems:[{Device:/run/containers/storage/overlay-containers/078982db28e53cedcf7da768f2acb81a2b85fe4d1adebd0108a2e4df654b3e97/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/846abc03-803a-4e83-839c-0cab2a41dc7b/volumes/kubernetes.io~projected/kube-api-access-jmp28 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/320ca86c27ed0fe917deb0ee65397ed4907424442cd5c04805b687f9b4f67bd8/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/692caf64-a543-4631-83a9-ce22eb1ec34a/volumes/kubernetes.io~projected/kube-api-access-bfcs4 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d24d05fd14e7e1c494c6bd7e49bd3c845a00f5683987f4673c78e88b6d0433b5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55393f54d534e67231a95de364b85d4b146d885821c86b74237ec1e52a536e0a/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f07043a164ea019a76d5dbd052ef2aa49c0d68e192f3eb4ae1d3ca13669135c2/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/75d094ba-dfa9-48bc-8ae7-028003bae8a8/volumes/kubernetes.io~projected/kube-api-access-r2tsl DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~projected/kube-api-access-mbdc7 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c12d5c7d-b992-4093-8b4f-1bf058820213/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94af8b5f-285c-47fe-9f67-f3b20f1d03c5/volumes/kubernetes.io~projected/kube-api-access-grr2k DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15acdc0d25e984a78d2ffab17243ebb3cf89603d1d0e8e1a761afccb4ad456a7/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9fafd6c-f585-4d85-a956-b536ee09d521/volumes/kubernetes.io~projected/kube-api-access-bg2xd DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dfd026479ff2f5d70582479fe0d0a8687605e0e59a5c7d668b9e5c8b2aba247/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/51861d34-b375-4586-bc20-472ed805e809/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc2033ebd17c9fb96cc1f5c5fdff3db67dd79b4d5bdf30a36954ab86872ff942/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fbea288aa9bdac61688f2add187983fa6d490e1e48f54491c2e8217696bb09a/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10a00aa9-5c78-48fc-8baf-40efe65c9db6/volumes/kubernetes.io~projected/kube-api-access-8pq48 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~projected/kube-api-access-hqfvp DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/244eeae6692babbeb58b41caa5ee3ea14a71d45c6c0fb803b1f66e6305d0336f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9554504-e458-45eb-ac2f-fb8bd160802e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bba37d6d-78c6-4e30-a76a-16cf6c61f424/volumes/kubernetes.io~projected/kube-api-access-9z9gf DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46765500-9cd2-4c86-a0cb-046a98b6ea69/volumes/kubernetes.io~projected/kube-api-access-xlntm DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a7323316-cacf-4ca1-afb1-7b38b203dfd7/volumes/kubernetes.io~projected/kube-api-access-cn8f6 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c39c1521-1467-4a9c-a993-ef855ebe4a0a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/461614a38d17871fba18f370edd54bc112b85cc54e338e1cfafdaf73b9104750/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/858fcc1f-21e9-4e03-b22e-115a8377a8fc/volumes/kubernetes.io~projected/kube-api-access-bpm9d DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d76833e6-9f22-425e-b95f-c8989faf3e8b/volumes/kubernetes.io~projected/kube-api-access-wfxcw DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33d17845-e07f-43a7-b2bf-c73d741f180f/volumes/kubernetes.io~projected/kube-api-access-crdtm DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/469899a6-621d-4421-9879-eba5b3c63f1e/volumes/kubernetes.io~projected/kube-api-access-xrp25 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74ae25254492cc355564ce79749c14e0fbf57c335066aac25e85dc551f40231a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15acdc0d25e984a MacAddress:7e:23:82:b6:16:af Speed:10000 Mtu:8901} {Name:320ca86c27ed0fe MacAddress:96:b4:ce:53:ca:e6 Speed:10000 Mtu:8901} {Name:32f74f68865f7b5 MacAddress:da:27:73:44:da:2e Speed:10000 Mtu:8901} {Name:55393f54d534e67 MacAddress:c6:f2:eb:af:fa:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dc:0f:7d:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:fb:d9:24:16:6c Speed:0 Mtu:8901} {Name:dc2033ebd17c9fb MacAddress:46:38:02:db:9d:d0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dc:0f:7d:b7 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:32:57:1e:1b:ff:9e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:5c:d7:3e:fb:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296295 4486 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296336 4486 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296650 4486 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296679 4486 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-114.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296792 4486 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296799 4486 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296809 4486 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296829 4486 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.296907 4486 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297059 4486 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297165 4486 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297197 4486 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297206 4486 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297214 4486 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297223 4486 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297230 4486 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297763 4486 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.297773 4486 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:11.299207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.298203 4486 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:57:11.300299 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300285 4486 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:11.300299 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300298 4486 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:11.300397 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300388 4486 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 00:57:11.300574 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300564 4486 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:57:11.300604 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300592 4486 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:57:11.300681 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300672 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:57:11.300713 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300686 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:57:11.300713 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300692 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:57:11.300713 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300697 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:57:11.300713 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300704 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:57:11.300713 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300711 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300721 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300730 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300740 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300747 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300770 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300786 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300837 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:57:11.300843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.300843 4486 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:57:11.304077 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.304065 4486 server.go:1263] "Started kubelet" Nov 04 00:57:11.304181 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.304148 4486 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:57:11.304314 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.304162 4486 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:57:11.304372 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.304338 4486 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:57:11.304751 ip-10-0-129-114 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:57:11.304907 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.304774 4486 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:57:11.305826 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.305808 4486 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:57:11.307601 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.307579 4486 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:57:11.308687 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.308656 4486 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.308923 4486 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.308955 4486 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:51:31 +0000 UTC" deadline="2028-05-18 11:26:27.228539106 +0000 UTC" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.309025 4486 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.309030 4486 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13474h29m15.919514139s" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.309034 4486 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.309046 4486 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311196 4486 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311210 4486 factory.go:55] Registering systemd factory Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311220 4486 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311815 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d17845-e07f-43a7-b2bf-c73d741f180f" volumeName="kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311854 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846abc03-803a-4e83-839c-0cab2a41dc7b" volumeName="kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311870 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7323316-cacf-4ca1-afb1-7b38b203dfd7" volumeName="kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311891 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9554504-e458-45eb-ac2f-fb8bd160802e" volumeName="kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311904 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692caf64-a543-4631-83a9-ce22eb1ec34a" volumeName="kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311923 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d094ba-dfa9-48bc-8ae7-028003bae8a8" volumeName="kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311946 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d094ba-dfa9-48bc-8ae7-028003bae8a8" volumeName="kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311961 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a00aa9-5c78-48fc-8baf-40efe65c9db6" volumeName="kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.311999 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a00aa9-5c78-48fc-8baf-40efe65c9db6" volumeName="kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312020 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51861d34-b375-4586-bc20-472ed805e809" volumeName="kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312040 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba37d6d-78c6-4e30-a76a-16cf6c61f424" volumeName="kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312056 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76833e6-9f22-425e-b95f-c8989faf3e8b" volumeName="kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312074 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76833e6-9f22-425e-b95f-c8989faf3e8b" volumeName="kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312087 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d17845-e07f-43a7-b2bf-c73d741f180f" volumeName="kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312112 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846abc03-803a-4e83-839c-0cab2a41dc7b" volumeName="kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312126 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858fcc1f-21e9-4e03-b22e-115a8377a8fc" volumeName="kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312142 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba37d6d-78c6-4e30-a76a-16cf6c61f424" volumeName="kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312157 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c1521-1467-4a9c-a993-ef855ebe4a0a" volumeName="kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312172 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c1521-1467-4a9c-a993-ef855ebe4a0a" volumeName="kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312192 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fafd6c-f585-4d85-a956-b536ee09d521" volumeName="kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312206 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75d094ba-dfa9-48bc-8ae7-028003bae8a8" volumeName="kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312237 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7323316-cacf-4ca1-afb1-7b38b203dfd7" volumeName="kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312251 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7323316-cacf-4ca1-afb1-7b38b203dfd7" volumeName="kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312266 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c39c1521-1467-4a9c-a993-ef855ebe4a0a" volumeName="kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312283 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fafd6c-f585-4d85-a956-b536ee09d521" volumeName="kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312296 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fafd6c-f585-4d85-a956-b536ee09d521" volumeName="kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312313 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51861d34-b375-4586-bc20-472ed805e809" volumeName="kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312325 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10a00aa9-5c78-48fc-8baf-40efe65c9db6" volumeName="kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312349 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846abc03-803a-4e83-839c-0cab2a41dc7b" volumeName="kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312832 4486 factory.go:153] Registering CRI-O factory Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312849 4486 factory.go:223] Registration of the crio container factory successfully Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312853 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858fcc1f-21e9-4e03-b22e-115a8377a8fc" volumeName="kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312868 4486 factory.go:103] Registering Raw factory Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312884 4486 manager.go:1199] Started watching for new ooms in manager Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312882 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94af8b5f-285c-47fe-9f67-f3b20f1d03c5" volumeName="kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312898 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7323316-cacf-4ca1-afb1-7b38b203dfd7" volumeName="kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312913 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bba37d6d-78c6-4e30-a76a-16cf6c61f424" volumeName="kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312933 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c12d5c7d-b992-4093-8b4f-1bf058820213" volumeName="kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312945 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d76833e6-9f22-425e-b95f-c8989faf3e8b" volumeName="kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312963 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46765500-9cd2-4c86-a0cb-046a98b6ea69" volumeName="kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.312975 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="469899a6-621d-4421-9879-eba5b3c63f1e" volumeName="kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313006 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692caf64-a543-4631-83a9-ce22eb1ec34a" volumeName="kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313022 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="846abc03-803a-4e83-839c-0cab2a41dc7b" volumeName="kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313034 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7323316-cacf-4ca1-afb1-7b38b203dfd7" volumeName="kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313050 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9fafd6c-f585-4d85-a956-b536ee09d521" volumeName="kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313062 4486 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9554504-e458-45eb-ac2f-fb8bd160802e" volumeName="kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs" seLinuxMountContext="" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313075 4486 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:57:11.313037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313082 4486 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:57:11.315408 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.313246 4486 manager.go:319] Starting recovery of all containers Nov 04 00:57:11.329706 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.329685 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:57:11.330637 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.330618 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:57:11.339702 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.339674 4486 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:57:11.340859 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.340838 4486 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:57:11.340859 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.340858 4486 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:57:11.341021 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.340877 4486 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:57:11.341021 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.340922 4486 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 00:57:11.347164 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347144 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="0868f1304c9c3008a5537cb76841654e0a12f9dac999300f0566c53bdacd2370" exitCode=0 Nov 04 00:57:11.347164 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347163 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="79b43ac394fee4a7d24fe3947e45f482dbbe99d2bd64e617115df28f0487f559" exitCode=0 Nov 04 00:57:11.347268 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347169 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="2674d555fa3d226606515341407c121732149e1a87de59f9100d28f925d0e699" exitCode=0 Nov 04 00:57:11.347268 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347176 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="a22fd9e985453fa9c0012117fc8079c62ad3548661502680cd9a7dff60ec53e2" exitCode=0 Nov 04 00:57:11.347268 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347181 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="0ecd94ee76e7b75f1297145203dc332041ae67fb16f01093b4bb0f335dd26b9d" exitCode=0 Nov 04 00:57:11.347268 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.347186 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9fafd6c-f585-4d85-a956-b536ee09d521" containerID="a188bed8ad3576acba6c894a0453d45ea78d10f6dc81fca048471e03013da2a9" exitCode=0 Nov 04 00:57:11.351865 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.351849 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 00:57:11.352194 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.352176 4486 generic.go:356] "Generic (PLEG): container finished" podID="a7323316-cacf-4ca1-afb1-7b38b203dfd7" containerID="c5378304da37916f95824213c5267670e4094177974188621bc636fa339a1878" exitCode=1 Nov 04 00:57:11.354693 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.354672 4486 generic.go:356] "Generic (PLEG): container finished" podID="5a400870bd799b9a4f8d0b646d2dfe4c" containerID="af8bdba16fa2ee5dc98d9540c0ee56307ff158c05b56dc8eb5903a46d68f63c2" exitCode=0 Nov 04 00:57:11.388651 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.388629 4486 manager.go:324] Recovery completed Nov 04 00:57:11.400955 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.400934 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.402019 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.401978 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.402106 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.402032 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.402106 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.402047 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.404029 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404007 4486 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:57:11.404029 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404025 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.404145 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404026 4486 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:57:11.404145 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404097 4486 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:57:11.404243 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404210 4486 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 00:57:11.404243 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404223 4486 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 00:57:11.404243 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404240 4486 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 00:57:11.404374 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404246 4486 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 00:57:11.404374 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404251 4486 policy_none.go:50] "Start" Nov 04 00:57:11.404374 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404264 4486 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:57:11.404374 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404273 4486 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:57:11.404547 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404388 4486 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 00:57:11.404547 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404395 4486 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 00:57:11.404547 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404400 4486 policy_none.go:44] "Start" Nov 04 00:57:11.404887 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404872 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.404941 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404903 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.404941 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.404913 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.408371 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.408348 4486 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:57:11.409047 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.409029 4486 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:57:11.411015 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.410999 4486 manager.go:341] "Starting Device Plugin manager" Nov 04 00:57:11.411091 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.411028 4486 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:57:11.411091 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411039 4486 server.go:87] "Starting device plugin registration server" Nov 04 00:57:11.411321 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411308 4486 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:57:11.411370 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411322 4486 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:57:11.412312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411429 4486 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:57:11.412312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411504 4486 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:57:11.412312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411531 4486 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:57:11.412312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411538 4486 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:57:11.412312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.411576 4486 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:57:11.411523158Z","UUID":"7da60087-84e5-4e05-a53d-69f94869460d","Handler":null,"Name":"","Endpoint":""} Nov 04 00:57:11.415111 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.415082 4486 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:57:11.415207 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.415123 4486 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:57:11.415491 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.415415 4486 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:57:11.415491 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.415454 4486 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" Nov 04 00:57:11.442040 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.441532 4486 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal"] Nov 04 00:57:11.442114 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.442087 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.442918 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.442900 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.442999 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.442930 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.442999 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.442943 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.443190 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443179 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.443328 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443316 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.443865 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443836 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.443925 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443867 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.443925 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443883 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.443925 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443903 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.443925 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443921 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.444082 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.443933 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.444082 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.444062 4486 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.444240 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444210 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerStarted","Data":"1b12e4d4cb71720ffdb0e4b7830481b53a1610d11bfa2ca822410acccca6f632"} Nov 04 00:57:11.444277 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444253 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerDied","Data":"af8bdba16fa2ee5dc98d9540c0ee56307ff158c05b56dc8eb5903a46d68f63c2"} Nov 04 00:57:11.444277 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444273 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"5a400870bd799b9a4f8d0b646d2dfe4c","Type":"ContainerStarted","Data":"23ad7a49b13344d90a26328395b89d04d8d7c1a1caca31d7b51fc7dcff20bf37"} Nov 04 00:57:11.444341 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444305 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.444341 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444307 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"93ea24cc66811a8502960a15b9987f67","Type":"ContainerStarted","Data":"405b4e60fcdf19e8e146e6fff7f55222d8043b60ae6fe17876d275c908d3a4c5"} Nov 04 00:57:11.444341 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.444326 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" event={"ID":"93ea24cc66811a8502960a15b9987f67","Type":"ContainerStarted","Data":"48254c614d224fd7322a74df518f8d7d3e380dfcb6a1019b465acabcb8865533"} Nov 04 00:57:11.445053 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.445038 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.445124 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.445067 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.445124 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.445082 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.445241 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.445228 4486 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-114.us-west-2.compute.internal\" not found" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.499626 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.499602 4486 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:11.499626 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.499615 4486 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:11.499734 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.499671 4486 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 00:57:11.511678 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.511663 4486 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:11.512481 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.512468 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:11.512538 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.512495 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:11.512538 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.512506 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:11.512538 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.512534 4486 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.546973 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.546952 4486 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:11.548037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.548018 4486 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:11.600802 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.600781 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.600914 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.600810 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.600914 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.600828 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.609606 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.609590 4486 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.615785 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.615769 4486 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.615843 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.615818 4486 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.616262 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.616248 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeNotReady" Nov 04 00:57:11.616314 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.616275 4486 setters.go:546] "Node became not ready" node="ip-10-0-129-114.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-04T00:57:11Z","lastTransitionTime":"2026-11-04T00:57:11Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 04 00:57:11.654266 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.654242 4486 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-114.us-west-2.compute.internal" is owned by different node (oldNodeID="d15d7e5e-7cb9-4dbf-8e9a-521abaf8ead8", newNodeID=""), cleaning up... Nov 04 00:57:11.667664 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.667647 4486 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:11.667722 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.667682 4486 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.667722 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.667694 4486 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701158 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701108 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701158 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701135 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701158 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701152 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701304 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701189 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93ea24cc66811a8502960a15b9987f67-config\") pod \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"93ea24cc66811a8502960a15b9987f67\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701304 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701203 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.701304 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.701217 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a400870bd799b9a4f8d0b646d2dfe4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" (UID: \"5a400870bd799b9a4f8d0b646d2dfe4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:11.766876 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:11.766856 4486 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:11.766948 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:11.766897 4486 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:12.297722 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.297698 4486 apiserver.go:52] "Watching apiserver" Nov 04 00:57:12.380351 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.380327 4486 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:12.380460 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.380448 4486 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:12.396816 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.396789 4486 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:57:12.398163 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.398130 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-slvp5","kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-lmxws","openshift-cluster-node-tuning-operator/tuned-bkc2z","openshift-insights/periodic-gathering-g2qlf-wb4jb","openshift-multus/multus-ls9l7","kube-system/konnectivity-agent-8knts","openshift-dns/dns-default-drlm7","openshift-ovn-kubernetes/ovnkube-node-9dx8l","openshift-image-registry/node-ca-t5scf","openshift-dns/node-resolver-7g22g","openshift-network-diagnostics/network-check-target-56frc","openshift-network-operator/iptables-alerter-zs49x","openshift-multus/multus-additional-cni-plugins-g69bf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b","openshift-insights/insights-runtime-extractor-nsgcc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-lkssf","openshift-service-ca/service-ca-5fffc97bdf-87hnt"] Nov 04 00:57:12.402472 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.402449 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.403288 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.403269 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.404116 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.404096 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.410187 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.410169 4486 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:57:12.438069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.438047 4486 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:12.438135 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:12.438090 4486 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:12.439101 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.439084 4486 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:12.439175 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:12.439116 4486 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-114.us-west-2.compute.internal" Nov 04 00:57:12.449956 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.449932 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.449956 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.449945 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.450106 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.449952 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:57:12.450106 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450047 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6tf45\"" Nov 04 00:57:12.450238 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450212 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.450510 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450487 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.450690 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450674 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450669 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450693 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gczmn\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450722 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450741 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450760 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.450769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450765 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:57:12.450955 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450789 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:57:12.450955 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450793 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.450955 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450886 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.450955 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.450903 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tdv96\"" Nov 04 00:57:12.451118 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451088 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:57:12.451118 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451110 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2w2b9\"" Nov 04 00:57:12.451182 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451143 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.451312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451285 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:57:12.451312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451298 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.451463 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451425 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ql7fx\"" Nov 04 00:57:12.451508 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451487 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xq5nc\"" Nov 04 00:57:12.451695 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451682 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qxmlz\"" Nov 04 00:57:12.451928 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.451916 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:12.504096 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.504072 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:57:12.504180 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.504153 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.504180 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.504150 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.504271 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.504189 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 00:57:12.504271 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.504171 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:57:12.505428 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505409 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqfvp\" (UniqueName: \"kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.505467 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505447 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.505515 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505467 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.505515 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505488 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.505614 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505535 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.505614 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505571 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.505614 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505612 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505614 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505622 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505679 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505689 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505650 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.505728 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505722 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505722 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505738 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505770 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505800 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505831 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505851 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505878 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505904 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505928 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.506010 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.505969 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506080 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9554504-e458-45eb-ac2f-fb8bd160802e-konnectivity-ca\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506103 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506146 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506153 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-daemon-config\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506107 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506180 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506231 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506272 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506313 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpm9d\" (UniqueName: \"kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506331 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506352 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9554504-e458-45eb-ac2f-fb8bd160802e-agent-certs\") pod \"konnectivity-agent-8knts\" (UID: \"f9554504-e458-45eb-ac2f-fb8bd160802e\") " pod="kube-system/konnectivity-agent-8knts" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506382 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.506448 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506410 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.506953 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506410 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.506953 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506905 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506953 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506912 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cni-binary-copy\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.506953 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.506949 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507013 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507048 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507080 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507117 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507134 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:57:12.507179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507150 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.507431 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507184 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:12.507431 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507245 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.507523 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507441 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.507523 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507469 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:57:12.507523 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507477 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.507523 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507509 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba37d6d-78c6-4e30-a76a-16cf6c61f424-data-volume\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.507707 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507511 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.507707 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507573 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.507707 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507684 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.507846 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507480 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10a00aa9-5c78-48fc-8baf-40efe65c9db6-tmp-dir\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.507846 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507807 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.507918 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507856 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.507963 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507920 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.508029 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.507977 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.508140 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508078 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.508140 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508124 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.508140 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508132 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.508282 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508157 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.508282 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508224 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z9gf\" (UniqueName: \"kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.508386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508302 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.508386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508338 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq48\" (UniqueName: \"kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.508386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508361 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg2xd\" (UniqueName: \"kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.508386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508367 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9fafd6c-f585-4d85-a956-b536ee09d521-cni-binary-copy\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.508386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508378 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508400 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508420 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tsl\" (UniqueName: \"kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508443 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508461 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdc7\" (UniqueName: \"kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508486 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508512 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8f6\" (UniqueName: \"kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508532 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.508619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508578 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.509019 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508860 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.509019 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.508997 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c12d5c7d-b992-4093-8b4f-1bf058820213-original-pull-secret\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.509211 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509189 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:12.509323 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509249 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.509323 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509290 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.509323 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509320 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.509474 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509376 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.509474 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509409 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrp25\" (UniqueName: \"kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.509557 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509475 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.509557 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509502 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlntm\" (UniqueName: \"kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.509643 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509569 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.509643 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509601 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.509643 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509630 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692caf64-a543-4631-83a9-ce22eb1ec34a-tmp-dir\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.509769 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509733 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.509886 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509865 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdtm\" (UniqueName: \"kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.509977 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510208 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510227 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510246 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510248 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-cabundle\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510285 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510318 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510817 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10a00aa9-5c78-48fc-8baf-40efe65c9db6-config-volume\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510862 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510893 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510922 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510950 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.510978 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511033 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511063 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511091 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511122 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.511228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511149 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511265 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511325 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511405 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511472 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511506 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511535 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511599 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511656 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511728 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-tuned\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511742 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511769 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511784 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d76833e6-9f22-425e-b95f-c8989faf3e8b-tmp\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511786 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511838 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511826 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33d17845-e07f-43a7-b2bf-c73d741f180f-serviceca\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511919 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.511964 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.512069 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512038 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512102 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxcw\" (UniqueName: \"kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512148 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512198 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcs4\" (UniqueName: \"kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512218 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-script-lib\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512244 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512279 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512313 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.512677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.512462 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/858fcc1f-21e9-4e03-b22e-115a8377a8fc-iptables-alerter-script\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.515940 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.515923 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-env-overrides\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.516048 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.515950 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovnkube-config\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.546806 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.546787 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fh9wk\"" Nov 04 00:57:12.547167 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.547141 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 00:57:12.551251 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.551209 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-47hrz\"" Nov 04 00:57:12.551443 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.551427 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-g2z88\"" Nov 04 00:57:12.551507 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.551433 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7knl\"" Nov 04 00:57:12.552877 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.552859 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c39c1521-1467-4a9c-a993-ef855ebe4a0a-signing-key\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.553109 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553087 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:12.553109 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553095 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553124 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8chh7\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553137 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553145 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553150 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wmbbs\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553154 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4d9rj\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553200 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-66h22\"" Nov 04 00:57:12.553242 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553210 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:12.553442 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553271 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6gr6q\"" Nov 04 00:57:12.553442 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.553283 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lbk4\"" Nov 04 00:57:12.556688 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.556672 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7323316-cacf-4ca1-afb1-7b38b203dfd7-ovn-node-metrics-cert\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.558537 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.558521 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51861d34-b375-4586-bc20-472ed805e809-cert\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:12.560054 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.560038 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpm9d\" (UniqueName: \"kubernetes.io/projected/858fcc1f-21e9-4e03-b22e-115a8377a8fc-kube-api-access-bpm9d\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.560116 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.560070 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqfvp\" (UniqueName: \"kubernetes.io/projected/c39c1521-1467-4a9c-a993-ef855ebe4a0a-kube-api-access-hqfvp\") pod \"service-ca-5fffc97bdf-87hnt\" (UID: \"c39c1521-1467-4a9c-a993-ef855ebe4a0a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-87hnt" Nov 04 00:57:12.560620 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.560605 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z9gf\" (UniqueName: \"kubernetes.io/projected/bba37d6d-78c6-4e30-a76a-16cf6c61f424-kube-api-access-9z9gf\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.560668 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.560655 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq48\" (UniqueName: \"kubernetes.io/projected/10a00aa9-5c78-48fc-8baf-40efe65c9db6-kube-api-access-8pq48\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.561249 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.561233 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg2xd\" (UniqueName: \"kubernetes.io/projected/e9fafd6c-f585-4d85-a956-b536ee09d521-kube-api-access-bg2xd\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.561977 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.561959 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tsl\" (UniqueName: \"kubernetes.io/projected/75d094ba-dfa9-48bc-8ae7-028003bae8a8-kube-api-access-r2tsl\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.562241 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.562225 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrp25\" (UniqueName: \"kubernetes.io/projected/469899a6-621d-4421-9879-eba5b3c63f1e-kube-api-access-xrp25\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.562301 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.562287 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8f6\" (UniqueName: \"kubernetes.io/projected/a7323316-cacf-4ca1-afb1-7b38b203dfd7-kube-api-access-cn8f6\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.562374 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.562360 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grr2k\" (UniqueName: \"kubernetes.io/projected/94af8b5f-285c-47fe-9f67-f3b20f1d03c5-kube-api-access-grr2k\") pod \"network-check-target-56frc\" (UID: \"94af8b5f-285c-47fe-9f67-f3b20f1d03c5\") " pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:12.562997 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.562968 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdc7\" (UniqueName: \"kubernetes.io/projected/51861d34-b375-4586-bc20-472ed805e809-kube-api-access-mbdc7\") pod \"ingress-canary-lmxws\" (UID: \"51861d34-b375-4586-bc20-472ed805e809\") " pod="openshift-ingress-canary/ingress-canary-lmxws" Nov 04 00:57:12.563873 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.563848 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlntm\" (UniqueName: \"kubernetes.io/projected/46765500-9cd2-4c86-a0cb-046a98b6ea69-kube-api-access-xlntm\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.563970 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.563953 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"periodic-gathering-g2qlf-wb4jb\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:12.564179 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.564161 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcs4\" (UniqueName: \"kubernetes.io/projected/692caf64-a543-4631-83a9-ce22eb1ec34a-kube-api-access-bfcs4\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.564416 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.564402 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxcw\" (UniqueName: \"kubernetes.io/projected/d76833e6-9f22-425e-b95f-c8989faf3e8b-kube-api-access-wfxcw\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.564802 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.564785 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdtm\" (UniqueName: \"kubernetes.io/projected/33d17845-e07f-43a7-b2bf-c73d741f180f-kube-api-access-crdtm\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.612898 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612878 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.612965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612904 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.612965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612918 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.612965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612932 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.612965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612946 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.612969 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613002 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-bin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613005 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-systemd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613027 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613034 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692caf64-a543-4631-83a9-ce22eb1ec34a-hosts-file\") pod \"node-resolver-7g22g\" (UID: \"692caf64-a543-4631-83a9-ce22eb1ec34a\") " pod="openshift-dns/node-resolver-7g22g" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613052 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613069 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613044 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-node-log\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613088 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-registration-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613074 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613052 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-system-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613103 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-cnibin\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613101 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bba37d6d-78c6-4e30-a76a-16cf6c61f424-crio-socket\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.613129 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613144 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613160 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613175 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613194 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613215 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613225 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-netns\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613229 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-kubelet\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613238 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:12.613253 4486 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613259 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613264 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysconfig\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613273 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613291 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613296 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-ovn\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613303 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-systemd\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613312 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613328 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:12.613338 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:13.113320144 +0000 UTC m=+1.871982303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613341 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613330 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33d17845-e07f-43a7-b2bf-c73d741f180f-host\") pod \"node-ca-t5scf\" (UID: \"33d17845-e07f-43a7-b2bf-c73d741f180f\") " pod="openshift-image-registry/node-ca-t5scf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613349 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-conf-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613358 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-system-cni-dir\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613377 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613396 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613411 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613421 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613446 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-cni-multus\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613450 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613426 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-etc-kubernetes\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613471 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613488 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613503 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613512 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-slash\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613517 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613535 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-log-socket\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613558 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-hostroot\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613557 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613565 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-lib-modules\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.613562 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613577 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613592 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613593 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-kubelet-config\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613607 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613621 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613635 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613650 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613669 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613678 4486 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613688 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-run-netns\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613694 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613690 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-sys-fs\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613702 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9fafd6c-f585-4d85-a956-b536ee09d521-cnibin\") pod \"multus-additional-cni-plugins-g69bf\" (UID: \"e9fafd6c-f585-4d85-a956-b536ee09d521\") " pod="openshift-multus/multus-additional-cni-plugins-g69bf" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613719 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613727 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-bin\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613743 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-etc-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613743 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613768 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-var-lib-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613718 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-k8s-cni-cncf-io\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613785 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613811 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613817 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-var-lib-kubelet\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613826 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-host-run-multus-certs\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613834 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613827 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-cni-dir\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613867 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613858 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-systemd-units\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613891 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613893 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-run-openvswitch\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613913 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613930 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-run\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613932 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613959 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613965 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-modprobe-d\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613972 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-host\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.613976 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614026 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614041 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-var-lib-kubelet\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614060 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614030 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-etc-selinux\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614083 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614087 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-sys\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614116 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858fcc1f-21e9-4e03-b22e-115a8377a8fc-host-slash\") pod \"iptables-alerter-zs49x\" (UID: \"858fcc1f-21e9-4e03-b22e-115a8377a8fc\") " pod="openshift-network-operator/iptables-alerter-zs49x" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614118 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75d094ba-dfa9-48bc-8ae7-028003bae8a8-multus-socket-dir-parent\") pod \"multus-ls9l7\" (UID: \"75d094ba-dfa9-48bc-8ae7-028003bae8a8\") " pod="openshift-multus/multus-ls9l7" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614135 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c12d5c7d-b992-4093-8b4f-1bf058820213-dbus\") pod \"global-pull-secret-syncer-lkssf\" (UID: \"c12d5c7d-b992-4093-8b4f-1bf058820213\") " pod="kube-system/global-pull-secret-syncer-lkssf" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614154 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614182 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614203 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614228 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614235 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-device-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614250 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614241 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-cni-netd\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614283 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614289 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7323316-cacf-4ca1-afb1-7b38b203dfd7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dx8l\" (UID: \"a7323316-cacf-4ca1-afb1-7b38b203dfd7\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614307 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-kubernetes\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614355 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46765500-9cd2-4c86-a0cb-046a98b6ea69-socket-dir\") pod \"aws-ebs-csi-driver-node-tql8b\" (UID: \"46765500-9cd2-4c86-a0cb-046a98b6ea69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tql8b" Nov 04 00:57:12.615150 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.614398 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d76833e6-9f22-425e-b95f-c8989faf3e8b-etc-sysctl-conf\") pod \"tuned-bkc2z\" (UID: \"d76833e6-9f22-425e-b95f-c8989faf3e8b\") " pod="openshift-cluster-node-tuning-operator/tuned-bkc2z" Nov 04 00:57:12.617571 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.616138 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469899a6-621d-4421-9879-eba5b3c63f1e-metrics-certs\") pod \"network-metrics-daemon-slvp5\" (UID: \"469899a6-621d-4421-9879-eba5b3c63f1e\") " pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.617571 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.616307 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bba37d6d-78c6-4e30-a76a-16cf6c61f424-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nsgcc\" (UID: \"bba37d6d-78c6-4e30-a76a-16cf6c61f424\") " pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:12.702717 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.702693 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-slvp5" Nov 04 00:57:12.704959 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:12.704944 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nsgcc" Nov 04 00:57:13.118521 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.118487 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:13.118674 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:13.118631 4486 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:13.118738 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:13.118701 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:14.118681222 +0000 UTC m=+2.877343379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:13.142479 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.142454 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-slvp5"] Nov 04 00:57:13.145897 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:13.145868 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469899a6_621d_4421_9879_eba5b3c63f1e.slice/crio-d0dc0cee51e0058c30c7ef54755f3239065156505bd9090fef32fc9b2ce63b94 WatchSource:0}: Error finding container d0dc0cee51e0058c30c7ef54755f3239065156505bd9090fef32fc9b2ce63b94: Status 404 returned error can't find the container with id d0dc0cee51e0058c30c7ef54755f3239065156505bd9090fef32fc9b2ce63b94 Nov 04 00:57:13.148193 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.148175 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:13.181347 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.181323 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nsgcc"] Nov 04 00:57:13.183647 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:13.183624 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba37d6d_78c6_4e30_a76a_16cf6c61f424.slice/crio-8735352d8dfdaa6a12ca61016374d1ddf9238308c967782e9020d42e1aedac80 WatchSource:0}: Error finding container 8735352d8dfdaa6a12ca61016374d1ddf9238308c967782e9020d42e1aedac80: Status 404 returned error can't find the container with id 8735352d8dfdaa6a12ca61016374d1ddf9238308c967782e9020d42e1aedac80 Nov 04 00:57:13.387362 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.387299 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsgcc" event={"ID":"bba37d6d-78c6-4e30-a76a-16cf6c61f424","Type":"ContainerStarted","Data":"e80bd8e03e5ee61db638dd1525a3f15d0f8a402b4d1723596fe7e5cc21ff3f64"} Nov 04 00:57:13.387738 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.387374 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsgcc" event={"ID":"bba37d6d-78c6-4e30-a76a-16cf6c61f424","Type":"ContainerStarted","Data":"8735352d8dfdaa6a12ca61016374d1ddf9238308c967782e9020d42e1aedac80"} Nov 04 00:57:13.388485 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:13.388460 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-slvp5" event={"ID":"469899a6-621d-4421-9879-eba5b3c63f1e","Type":"ContainerStarted","Data":"d0dc0cee51e0058c30c7ef54755f3239065156505bd9090fef32fc9b2ce63b94"} Nov 04 00:57:14.124579 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:14.124543 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:14.124735 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:14.124687 4486 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:14.124771 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:14.124751 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:16.124737647 +0000 UTC m=+4.883399800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:16.137513 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:16.137482 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:16.137931 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:16.137598 4486 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:16.137931 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:16.137644 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:20.137631201 +0000 UTC m=+8.896293340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:16.399505 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:16.399433 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsgcc" event={"ID":"bba37d6d-78c6-4e30-a76a-16cf6c61f424","Type":"ContainerStarted","Data":"bbd8ce2f7c695faf43676da84db7a7bcad110176dfe11c3e82f45b8fc75bfe26"} Nov 04 00:57:16.401142 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:16.401117 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-slvp5" event={"ID":"469899a6-621d-4421-9879-eba5b3c63f1e","Type":"ContainerStarted","Data":"92f279302153ba7ac1828a988028427a661770c4d9cc7c404f228f90243f2c9f"} Nov 04 00:57:16.401252 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:16.401158 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-slvp5" event={"ID":"469899a6-621d-4421-9879-eba5b3c63f1e","Type":"ContainerStarted","Data":"f85a2a5e33aa781d788b59585ef305921456d6f7a8375660233a594b7c562e66"} Nov 04 00:57:16.948426 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:16.948393 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:18.378267 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:18.378238 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8knts" Nov 04 00:57:18.409923 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:18.409893 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nsgcc" event={"ID":"bba37d6d-78c6-4e30-a76a-16cf6c61f424","Type":"ContainerStarted","Data":"1228faaf325cebf136589d0b1e27bbb59e5ca92182f9b1705d970235ca0b0c12"} Nov 04 00:57:18.897186 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:18.897159 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:19.859425 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:19.859391 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:19.859875 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:19.859580 4486 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:20.163862 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:20.163760 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:20.164042 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:20.163918 4486 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:20.164042 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:20.164030 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls podName:10a00aa9-5c78-48fc-8baf-40efe65c9db6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:28.164008157 +0000 UTC m=+16.922670309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls") pod "dns-default-drlm7" (UID: "10a00aa9-5c78-48fc-8baf-40efe65c9db6") : secret "dns-default-metrics-tls" not found Nov 04 00:57:21.934858 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:21.934827 4486 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-114.us-west-2.compute.internal" event="NodeReady" Nov 04 00:57:25.433569 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.433539 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f9f58767f-r9rwh"] Nov 04 00:57:25.435289 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.435271 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498162 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498132 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-tls\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498288 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498177 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-trusted-ca\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498339 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498283 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-image-registry-private-configuration\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498399 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498335 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ddbc7a-0e87-4005-beba-8a88ca2c2186-ca-trust-extracted\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498455 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498399 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-certificates\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498455 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498442 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-installation-pull-secrets\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498536 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498509 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-bound-sa-token\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.498536 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.498529 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrxp\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-kube-api-access-9vrxp\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.587794 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.587766 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 00:57:25.587794 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.587767 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 00:57:25.598941 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.598921 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-bound-sa-token\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599031 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.598946 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrxp\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-kube-api-access-9vrxp\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599031 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.598965 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-tls\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599142 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599105 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-trusted-ca\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599196 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599149 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-image-registry-private-configuration\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599248 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599203 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ddbc7a-0e87-4005-beba-8a88ca2c2186-ca-trust-extracted\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599248 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599228 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-certificates\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599345 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599260 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-installation-pull-secrets\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.599695 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.599670 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32ddbc7a-0e87-4005-beba-8a88ca2c2186-ca-trust-extracted\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.600040 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.600019 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-certificates\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.601584 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.601561 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-image-registry-private-configuration\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.601666 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.601604 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-registry-tls\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.635098 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.635076 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 00:57:25.635209 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.635099 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w95zk\"" Nov 04 00:57:25.635292 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.635271 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 00:57:25.635725 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.635707 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9f58767f-r9rwh"] Nov 04 00:57:25.640258 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.640239 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32ddbc7a-0e87-4005-beba-8a88ca2c2186-trusted-ca\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.641599 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.641578 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32ddbc7a-0e87-4005-beba-8a88ca2c2186-installation-pull-secrets\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.653479 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.653457 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-bound-sa-token\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.653725 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.653707 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrxp\" (UniqueName: \"kubernetes.io/projected/32ddbc7a-0e87-4005-beba-8a88ca2c2186-kube-api-access-9vrxp\") pod \"image-registry-6f9f58767f-r9rwh\" (UID: \"32ddbc7a-0e87-4005-beba-8a88ca2c2186\") " pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:25.745819 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:25.745765 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:26.202055 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:26.201845 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f9f58767f-r9rwh"] Nov 04 00:57:26.206120 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:26.206092 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ddbc7a_0e87_4005_beba_8a88ca2c2186.slice/crio-69a03835980b64529a0a55731012aaa7decc116890c324dda1476e077d01d2e7 WatchSource:0}: Error finding container 69a03835980b64529a0a55731012aaa7decc116890c324dda1476e077d01d2e7: Status 404 returned error can't find the container with id 69a03835980b64529a0a55731012aaa7decc116890c324dda1476e077d01d2e7 Nov 04 00:57:26.433648 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:26.433613 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" event={"ID":"32ddbc7a-0e87-4005-beba-8a88ca2c2186","Type":"ContainerStarted","Data":"0fa8a4b53be64db9ac356a8314d944fa01b7f81efd3a4a4306cb4f76c7aec56b"} Nov 04 00:57:26.434118 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:26.433666 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" event={"ID":"32ddbc7a-0e87-4005-beba-8a88ca2c2186","Type":"ContainerStarted","Data":"69a03835980b64529a0a55731012aaa7decc116890c324dda1476e077d01d2e7"} Nov 04 00:57:26.434118 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:26.433739 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:26.546410 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:26.546313 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" podStartSLOduration=1.546299213 podStartE2EDuration="1.546299213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:26.54612194 +0000 UTC m=+15.304784101" watchObservedRunningTime="2026-11-04 00:57:26.546299213 +0000 UTC m=+15.304961370" Nov 04 00:57:28.218551 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:28.218511 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:28.220841 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:28.220818 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10a00aa9-5c78-48fc-8baf-40efe65c9db6-metrics-tls\") pod \"dns-default-drlm7\" (UID: \"10a00aa9-5c78-48fc-8baf-40efe65c9db6\") " pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:28.303592 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:28.303562 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:28.663388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:28.663356 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-drlm7"] Nov 04 00:57:28.666556 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:28.666528 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a00aa9_5c78_48fc_8baf_40efe65c9db6.slice/crio-3d7e827e3be59e38b9107a70340b66376b1bcab2d9a1a1147195ec589a50e6a5 WatchSource:0}: Error finding container 3d7e827e3be59e38b9107a70340b66376b1bcab2d9a1a1147195ec589a50e6a5: Status 404 returned error can't find the container with id 3d7e827e3be59e38b9107a70340b66376b1bcab2d9a1a1147195ec589a50e6a5 Nov 04 00:57:29.445887 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:29.445852 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drlm7" event={"ID":"10a00aa9-5c78-48fc-8baf-40efe65c9db6","Type":"ContainerStarted","Data":"3d7e827e3be59e38b9107a70340b66376b1bcab2d9a1a1147195ec589a50e6a5"} Nov 04 00:57:31.453156 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:31.453117 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drlm7" event={"ID":"10a00aa9-5c78-48fc-8baf-40efe65c9db6","Type":"ContainerStarted","Data":"3e86c3792eeb3f13aab55b7b0c769855a4d839923fa00798f0d6727fa995cf14"} Nov 04 00:57:31.453542 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:31.453177 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-drlm7" event={"ID":"10a00aa9-5c78-48fc-8baf-40efe65c9db6","Type":"ContainerStarted","Data":"1e2b37f7529e96984c4ea3c543132282b77137b245f8b2386ed29da7d2c6b4c3"} Nov 04 00:57:31.453542 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:31.453224 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:35.901855 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.901817 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc"] Nov 04 00:57:35.904873 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.904856 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:35.964245 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.964212 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:35.964385 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.964269 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4716a-952a-4390-a02d-9628df25ef87-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:35.964385 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.964369 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/16f4716a-952a-4390-a02d-9628df25ef87-kube-api-access-dv7sk\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:35.964475 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:35.964417 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.002077 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002044 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:57:36.002188 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002079 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:57:36.002188 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002085 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-44sfv\"" Nov 04 00:57:36.002188 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002113 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 04 00:57:36.002188 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002149 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 04 00:57:36.002607 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.002586 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:57:36.050192 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.050163 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc"] Nov 04 00:57:36.064960 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.064939 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.065064 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.064972 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4716a-952a-4390-a02d-9628df25ef87-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.065064 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.065008 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/16f4716a-952a-4390-a02d-9628df25ef87-kube-api-access-dv7sk\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.065064 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.065030 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.065633 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.065607 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16f4716a-952a-4390-a02d-9628df25ef87-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.067304 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.067282 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.067473 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.067454 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16f4716a-952a-4390-a02d-9628df25ef87-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.120278 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.120249 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7sk\" (UniqueName: \"kubernetes.io/projected/16f4716a-952a-4390-a02d-9628df25ef87-kube-api-access-dv7sk\") pod \"prometheus-operator-9cd89b74c-fhcrc\" (UID: \"16f4716a-952a-4390-a02d-9628df25ef87\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.216751 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.216698 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" Nov 04 00:57:36.570362 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.570332 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dx8l" Nov 04 00:57:36.578383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:36.578355 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc"] Nov 04 00:57:36.581287 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:36.581265 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f4716a_952a_4390_a02d_9628df25ef87.slice/crio-ee6d06a6655596c141c4d91292872fc372b1354843a97bd65e47bcc82fd4c969 WatchSource:0}: Error finding container ee6d06a6655596c141c4d91292872fc372b1354843a97bd65e47bcc82fd4c969: Status 404 returned error can't find the container with id ee6d06a6655596c141c4d91292872fc372b1354843a97bd65e47bcc82fd4c969 Nov 04 00:57:37.474465 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:37.474428 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" event={"ID":"16f4716a-952a-4390-a02d-9628df25ef87","Type":"ContainerStarted","Data":"ee6d06a6655596c141c4d91292872fc372b1354843a97bd65e47bcc82fd4c969"} Nov 04 00:57:38.055632 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.055547 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-dqjq2-k5dll"] Nov 04 00:57:38.058309 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.058280 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.079599 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.079573 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.079724 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.079621 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.079724 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.079702 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79h6\" (UniqueName: \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.079833 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.079757 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.118228 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.118203 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dqjq2-k5dll"] Nov 04 00:57:38.180946 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.180911 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.181192 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.180964 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79h6\" (UniqueName: \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.181192 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.181015 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.181192 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.181066 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.181467 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.181449 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.181862 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.181840 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.182157 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.182135 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.236890 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.236855 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79h6\" (UniqueName: \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\") pod \"periodic-gathering-dqjq2-k5dll\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.371576 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.371546 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:57:38.749749 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:38.749722 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-dqjq2-k5dll"] Nov 04 00:57:38.757526 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:38.757467 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8608fb_515b_42c3_ae4f_6eaf612c2ae1.slice/crio-bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b WatchSource:0}: Error finding container bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b: Status 404 returned error can't find the container with id bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b Nov 04 00:57:39.483855 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.483815 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" event={"ID":"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1","Type":"ContainerStarted","Data":"fcabbe1f88e09616f964bdeea1ccddd6207c07c252120842928d51bb7068b228"} Nov 04 00:57:39.484075 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.483866 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" event={"ID":"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1","Type":"ContainerStarted","Data":"bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b"} Nov 04 00:57:39.485638 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.485612 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" event={"ID":"16f4716a-952a-4390-a02d-9628df25ef87","Type":"ContainerStarted","Data":"d0357c985acdab0d88d3377cfd4e6c51bfc3722c25ce37a3c383253e883d4bca"} Nov 04 00:57:39.485736 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.485652 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" event={"ID":"16f4716a-952a-4390-a02d-9628df25ef87","Type":"ContainerStarted","Data":"346b69f905700642d25e26c2679057c916d843490676408d3217825b08ab8456"} Nov 04 00:57:39.592745 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.592704 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" podStartSLOduration=1.592693135 podStartE2EDuration="1.592693135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:39.592065969 +0000 UTC m=+28.350728128" watchObservedRunningTime="2026-11-04 00:57:39.592693135 +0000 UTC m=+28.351355295" Nov 04 00:57:39.709318 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:39.709270 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-fhcrc" podStartSLOduration=2.620961517 podStartE2EDuration="4.709259059s" totalImagesPullingTime="2.088297542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:39.708916304 +0000 UTC m=+28.467578463" watchObservedRunningTime="2026-11-04 00:57:39.709259059 +0000 UTC m=+28.467921244" Nov 04 00:57:41.460488 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:41.460457 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-drlm7" Nov 04 00:57:41.898315 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:41.898282 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-drlm7_10a00aa9-5c78-48fc-8baf-40efe65c9db6/dns/0.log" Nov 04 00:57:42.009777 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.009745 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-drlm7_10a00aa9-5c78-48fc-8baf-40efe65c9db6/kube-rbac-proxy/0.log" Nov 04 00:57:42.501081 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.501052 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7g22g_692caf64-a543-4631-83a9-ce22eb1ec34a/dns-node-resolver/0.log" Nov 04 00:57:42.512831 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.512806 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kfb5n"] Nov 04 00:57:42.544942 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.544916 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610542 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610515 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-wtmp\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610657 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610557 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610657 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610574 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4t2\" (UniqueName: \"kubernetes.io/projected/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-kube-api-access-ct4t2\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610744 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610666 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-textfile\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610744 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610723 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-root\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610829 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610745 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610829 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610765 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-sys\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610829 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610779 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-tls\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.610829 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.610794 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-metrics-client-ca\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.641406 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.641387 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tjpbs\"" Nov 04 00:57:42.641499 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.641488 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:57:42.641541 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.641508 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:57:42.641583 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.641574 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:57:42.711243 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711220 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711247 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4t2\" (UniqueName: \"kubernetes.io/projected/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-kube-api-access-ct4t2\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711265 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-textfile\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711290 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-root\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711336 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-root\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711530 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711462 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711530 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711512 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-sys\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711600 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711540 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-tls\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711600 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711565 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-metrics-client-ca\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711691 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711599 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-wtmp\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711691 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711624 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-textfile\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711691 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711625 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-sys\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.711836 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.711762 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-wtmp\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.712032 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.712013 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.712135 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.712119 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-metrics-client-ca\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.713619 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.713602 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.713788 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.713768 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-node-exporter-tls\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.773006 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.772938 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4t2\" (UniqueName: \"kubernetes.io/projected/1a6fab7b-b666-4651-a2ef-9b4c5dbadf84-kube-api-access-ct4t2\") pod \"node-exporter-kfb5n\" (UID: \"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84\") " pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.895357 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:42.895326 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfb5n" Nov 04 00:57:42.926298 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:42.926267 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6fab7b_b666_4651_a2ef_9b4c5dbadf84.slice/crio-fa7e0428df2b0c117c6dc31249d97aeea78e5485a8625cdb12894d8ebce13a65 WatchSource:0}: Error finding container fa7e0428df2b0c117c6dc31249d97aeea78e5485a8625cdb12894d8ebce13a65: Status 404 returned error can't find the container with id fa7e0428df2b0c117c6dc31249d97aeea78e5485a8625cdb12894d8ebce13a65 Nov 04 00:57:43.490232 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:43.490206 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lmxws_51861d34-b375-4586-bc20-472ed805e809/serve-healthcheck-canary/0.log" Nov 04 00:57:43.498259 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:43.498233 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfb5n" event={"ID":"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84","Type":"ContainerStarted","Data":"fa7e0428df2b0c117c6dc31249d97aeea78e5485a8625cdb12894d8ebce13a65"} Nov 04 00:57:43.835321 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:43.833133 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 00:57:45.199383 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.199353 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68795d64cd-6f466"] Nov 04 00:57:45.206476 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.206460 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.230900 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.230844 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68795d64cd-6f466"] Nov 04 00:57:45.277094 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277065 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 04 00:57:45.277202 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277089 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 04 00:57:45.277202 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277157 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 04 00:57:45.277202 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277161 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 04 00:57:45.277350 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277257 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4a9ipu9bqg5rg\"" Nov 04 00:57:45.277388 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277345 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 04 00:57:45.277426 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.277399 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pkltv\"" Nov 04 00:57:45.330062 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-grpc-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330140 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330073 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330140 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330104 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef840115-544e-4e5b-b573-9938e28ecec7-metrics-client-ca\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330247 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330228 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330288 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330256 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330327 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330307 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330362 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330353 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330393 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330375 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef840115-544e-4e5b-b573-9938e28ecec7-tmp\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.330435 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.330421 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chtk\" (UniqueName: \"kubernetes.io/projected/ef840115-544e-4e5b-b573-9938e28ecec7-kube-api-access-9chtk\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431402 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431377 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431484 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431405 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef840115-544e-4e5b-b573-9938e28ecec7-tmp\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431484 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431433 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chtk\" (UniqueName: \"kubernetes.io/projected/ef840115-544e-4e5b-b573-9938e28ecec7-kube-api-access-9chtk\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431484 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431465 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-grpc-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431636 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431485 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431636 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431510 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef840115-544e-4e5b-b573-9938e28ecec7-metrics-client-ca\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431636 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431538 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431636 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431580 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431848 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431809 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef840115-544e-4e5b-b573-9938e28ecec7-tmp\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.431906 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.431868 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.432286 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.432260 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef840115-544e-4e5b-b573-9938e28ecec7-metrics-client-ca\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434226 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434201 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434308 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434212 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-grpc-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434449 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434428 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434484 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434436 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434555 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434534 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-tls\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.434832 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.434815 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef840115-544e-4e5b-b573-9938e28ecec7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.486683 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.486633 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chtk\" (UniqueName: \"kubernetes.io/projected/ef840115-544e-4e5b-b573-9938e28ecec7-kube-api-access-9chtk\") pod \"thanos-querier-68795d64cd-6f466\" (UID: \"ef840115-544e-4e5b-b573-9938e28ecec7\") " pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.506935 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.506911 4486 generic.go:356] "Generic (PLEG): container finished" podID="1a6fab7b-b666-4651-a2ef-9b4c5dbadf84" containerID="252e01c0677386c76e8572b5d06ff92d4e6e3a6d0e32cf4d651b3c3c6ae4f822" exitCode=0 Nov 04 00:57:45.507048 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.506953 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfb5n" event={"ID":"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84","Type":"ContainerDied","Data":"252e01c0677386c76e8572b5d06ff92d4e6e3a6d0e32cf4d651b3c3c6ae4f822"} Nov 04 00:57:45.541522 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.541503 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:45.749879 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.749808 4486 patch_prober.go:29] interesting pod/image-registry-6f9f58767f-r9rwh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 00:57:45.750009 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:45.749860 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" podUID="32ddbc7a-0e87-4005-beba-8a88ca2c2186" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 00:57:46.030631 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.030544 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68795d64cd-6f466"] Nov 04 00:57:46.033198 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:46.033165 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef840115_544e_4e5b_b573_9938e28ecec7.slice/crio-8a9201333c327f0c06be3652b2f8a5a0e6a52e828287f1e479f3883b0b12e2e2 WatchSource:0}: Error finding container 8a9201333c327f0c06be3652b2f8a5a0e6a52e828287f1e479f3883b0b12e2e2: Status 404 returned error can't find the container with id 8a9201333c327f0c06be3652b2f8a5a0e6a52e828287f1e479f3883b0b12e2e2 Nov 04 00:57:46.513410 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.513373 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfb5n" event={"ID":"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84","Type":"ContainerStarted","Data":"297f69ed0124b8c19329c118d4325efb03bd78573a5c93471b3a88a85636d5a9"} Nov 04 00:57:46.513860 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.513437 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfb5n" event={"ID":"1a6fab7b-b666-4651-a2ef-9b4c5dbadf84","Type":"ContainerStarted","Data":"2d7f76717875601c175ee6e5a29754bdd164ca24478b55d4d4fdb0d0ff2d116f"} Nov 04 00:57:46.514772 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.514727 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"8a9201333c327f0c06be3652b2f8a5a0e6a52e828287f1e479f3883b0b12e2e2"} Nov 04 00:57:46.624677 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.624622 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kfb5n" podStartSLOduration=2.50929174 podStartE2EDuration="4.624607843s" totalImagesPullingTime="2.040683223s" totalInitContainerRuntime="74.63288ms" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:46.623779566 +0000 UTC m=+35.382441728" watchObservedRunningTime="2026-11-04 00:57:46.624607843 +0000 UTC m=+35.383270004" Nov 04 00:57:46.951711 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:46.951679 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-56frc" Nov 04 00:57:47.196235 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.196199 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr"] Nov 04 00:57:47.198340 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.198323 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.245816 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.245787 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36270d33-26d2-4047-9db9-0d418dda1d02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.245937 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.245834 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstzh\" (UniqueName: \"kubernetes.io/projected/36270d33-26d2-4047-9db9-0d418dda1d02-kube-api-access-cstzh\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.253797 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.253772 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr"] Nov 04 00:57:47.271066 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.271032 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 04 00:57:47.271175 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.271112 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j7vb5\"" Nov 04 00:57:47.346570 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.346540 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36270d33-26d2-4047-9db9-0d418dda1d02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.346706 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.346615 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstzh\" (UniqueName: \"kubernetes.io/projected/36270d33-26d2-4047-9db9-0d418dda1d02-kube-api-access-cstzh\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.349286 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.349259 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/36270d33-26d2-4047-9db9-0d418dda1d02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.401206 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.401181 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstzh\" (UniqueName: \"kubernetes.io/projected/36270d33-26d2-4047-9db9-0d418dda1d02-kube-api-access-cstzh\") pod \"monitoring-plugin-77f4f76594-xmdnr\" (UID: \"36270d33-26d2-4047-9db9-0d418dda1d02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.441279 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.441249 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f9f58767f-r9rwh" Nov 04 00:57:47.508703 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.508640 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:47.881465 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:47.881432 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr"] Nov 04 00:57:47.885603 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:57:47.885568 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36270d33_26d2_4047_9db9_0d418dda1d02.slice/crio-6a0f6c2a8a7f961c9ffe8ee27e977b084a3ed9bcb29bf980ac9feadd1e2892d2 WatchSource:0}: Error finding container 6a0f6c2a8a7f961c9ffe8ee27e977b084a3ed9bcb29bf980ac9feadd1e2892d2: Status 404 returned error can't find the container with id 6a0f6c2a8a7f961c9ffe8ee27e977b084a3ed9bcb29bf980ac9feadd1e2892d2 Nov 04 00:57:48.523880 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:48.523841 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" event={"ID":"36270d33-26d2-4047-9db9-0d418dda1d02","Type":"ContainerStarted","Data":"6a0f6c2a8a7f961c9ffe8ee27e977b084a3ed9bcb29bf980ac9feadd1e2892d2"} Nov 04 00:57:49.529441 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:49.529406 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"8772ac1bd0d15f7c8733bed397c37ed8ea12e0f97efe687da98eca49612f902c"} Nov 04 00:57:49.529851 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:49.529508 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"90c0874dd9dfad2fb3fc4478d37711f3e9fee5db39bfb26977714a80290806a1"} Nov 04 00:57:49.529851 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:49.529538 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"4883860d4a5630ade7329732ff2c6505f0df3d55e2e71b0fb4ef8ed9c4f1058e"} Nov 04 00:57:50.536265 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.536187 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"e97ca98a013f2bc7a2462b5925aa3950048a15e8dc67075d3b406531681db55e"} Nov 04 00:57:50.536660 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.536266 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"db5f42ddcd91ba2326538b1ba831e89b878fe5b948a39deea41f2c1862e909fb"} Nov 04 00:57:50.536660 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.536311 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:50.536660 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.536365 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" event={"ID":"ef840115-544e-4e5b-b573-9938e28ecec7","Type":"ContainerStarted","Data":"7039e61c3eea2d692e4e177145e06bd70fd74e270691b9355b3cbe79c07c709e"} Nov 04 00:57:50.537735 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.537708 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" event={"ID":"36270d33-26d2-4047-9db9-0d418dda1d02","Type":"ContainerStarted","Data":"0f4c0fd334f0343647bd06abdb2e87230b1e3e16c3ebb829b70cc0d5624466d1"} Nov 04 00:57:50.537865 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.537849 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:50.542704 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.542681 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" Nov 04 00:57:50.791376 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.791288 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" podStartSLOduration=1.7975311029999999 podStartE2EDuration="5.791272841s" totalImagesPullingTime="3.993741738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:50.74330378 +0000 UTC m=+39.501965940" watchObservedRunningTime="2026-11-04 00:57:50.791272841 +0000 UTC m=+39.549935002" Nov 04 00:57:50.850294 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:50.850243 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-xmdnr" podStartSLOduration=1.915971394 podStartE2EDuration="3.850229195s" totalImagesPullingTime="1.934257801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:50.849511479 +0000 UTC m=+39.608173640" watchObservedRunningTime="2026-11-04 00:57:50.850229195 +0000 UTC m=+39.608891355" Nov 04 00:57:56.548241 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:56.548165 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68795d64cd-6f466" Nov 04 00:57:57.316349 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.316314 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:57.316852 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.316811 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" podUID="846abc03-803a-4e83-839c-0cab2a41dc7b" containerName="insights-gathering" containerID="cri-o://0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0" gracePeriod=30 Nov 04 00:57:57.553694 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.553670 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:57.560239 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.560211 4486 generic.go:356] "Generic (PLEG): container finished" podID="846abc03-803a-4e83-839c-0cab2a41dc7b" containerID="0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0" exitCode=2 Nov 04 00:57:57.560323 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.560247 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" Nov 04 00:57:57.560323 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.560284 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" event={"ID":"846abc03-803a-4e83-839c-0cab2a41dc7b","Type":"ContainerDied","Data":"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0"} Nov 04 00:57:57.560398 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.560327 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g2qlf-wb4jb" event={"ID":"846abc03-803a-4e83-839c-0cab2a41dc7b","Type":"ContainerDied","Data":"32f74f68865f7b58a6086c2f9bbbb08fb7df0c4fab157481a0cb24a263b8e563"} Nov 04 00:57:57.560398 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.560353 4486 scope.go:122] "RemoveContainer" containerID="0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0" Nov 04 00:57:57.571888 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.571859 4486 scope.go:122] "RemoveContainer" containerID="0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0" Nov 04 00:57:57.572210 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:57:57.572190 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0\": container with ID starting with 0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0 not found: ID does not exist" containerID="0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0" Nov 04 00:57:57.572271 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.572221 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0"} err="failed to get container status \"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0\": rpc error: code = NotFound desc = could not find container \"0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0\": container with ID starting with 0bb022f5e05adcb9e7e0e5abed8a77dc9a4d46de18f235646531a0d3e8db99b0 not found: ID does not exist" Nov 04 00:57:57.621465 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621435 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") pod \"846abc03-803a-4e83-839c-0cab2a41dc7b\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " Nov 04 00:57:57.621601 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621491 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") pod \"846abc03-803a-4e83-839c-0cab2a41dc7b\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " Nov 04 00:57:57.621601 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621543 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") pod \"846abc03-803a-4e83-839c-0cab2a41dc7b\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " Nov 04 00:57:57.621601 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621573 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") pod \"846abc03-803a-4e83-839c-0cab2a41dc7b\" (UID: \"846abc03-803a-4e83-839c-0cab2a41dc7b\") " Nov 04 00:57:57.621819 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621791 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path" pod "846abc03-803a-4e83-839c-0cab2a41dc7b" (UID: "846abc03-803a-4e83-839c-0cab2a41dc7b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:57:57.621894 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621871 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle" pod "846abc03-803a-4e83-839c-0cab2a41dc7b" (UID: "846abc03-803a-4e83-839c-0cab2a41dc7b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:57:57.621931 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.621909 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle" pod "846abc03-803a-4e83-839c-0cab2a41dc7b" (UID: "846abc03-803a-4e83-839c-0cab2a41dc7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:57:57.623692 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.623675 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28" pod "846abc03-803a-4e83-839c-0cab2a41dc7b" (UID: "846abc03-803a-4e83-839c-0cab2a41dc7b"). InnerVolumeSpecName "kube-api-access-jmp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:57:57.722213 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.722182 4486 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/846abc03-803a-4e83-839c-0cab2a41dc7b-archives-path\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:57.722213 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.722206 4486 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-service-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:57.722213 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.722216 4486 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/846abc03-803a-4e83-839c-0cab2a41dc7b-trusted-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:57.722401 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.722226 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmp28\" (UniqueName: \"kubernetes.io/projected/846abc03-803a-4e83-839c-0cab2a41dc7b-kube-api-access-jmp28\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:57:57.973386 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.973356 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:57.978241 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:57.978220 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-g2qlf-wb4jb"] Nov 04 00:57:59.345725 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:57:59.345691 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846abc03-803a-4e83-839c-0cab2a41dc7b" path="/var/lib/kubelet/pods/846abc03-803a-4e83-839c-0cab2a41dc7b/volumes" Nov 04 00:58:03.144595 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.144563 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:58:03.144966 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.144824 4486 cpu_manager.go:414] "containerMap: removing container" podUID="846abc03-803a-4e83-839c-0cab2a41dc7b" containerName="insights-gathering" Nov 04 00:58:03.144966 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.144834 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846abc03-803a-4e83-839c-0cab2a41dc7b" containerName="insights-gathering" Nov 04 00:58:03.144966 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.144893 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="846abc03-803a-4e83-839c-0cab2a41dc7b" containerName="insights-gathering" Nov 04 00:58:03.153361 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.153333 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.203037 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.203015 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6x79p\"" Nov 04 00:58:03.203143 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.203080 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 00:58:03.250366 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.250340 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 00:58:03.250458 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.250447 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 00:58:03.250659 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.250565 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 00:58:03.277533 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277505 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277620 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277556 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277671 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277625 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrz82\" (UniqueName: \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277716 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277696 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277773 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277760 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277822 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277805 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277900 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277879 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.277946 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.277933 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.297882 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.297827 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 00:58:03.298053 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.297962 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 00:58:03.298053 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.298042 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 00:58:03.346539 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.346520 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 00:58:03.347471 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.347455 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:58:03.378629 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378609 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.378719 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378644 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.378719 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378660 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrz82\" (UniqueName: \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.378868 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378844 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.378929 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378911 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379003 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378932 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379003 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378956 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379003 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.378974 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379160 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.379106 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379341 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.379320 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379637 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.379613 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379718 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.379690 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.379836 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.379819 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.381294 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.381274 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.381365 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.381326 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.432148 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.432090 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrz82\" (UniqueName: \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\") pod \"console-74c9c4bcdc-q9hwj\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.468062 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.468042 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:03.830444 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:03.830416 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:58:03.833401 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:58:03.833349 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53df385f_18fa_4ca3_a39a_0d5c0f4bc539.slice/crio-4b5f19ac92ae5427c430b52ffb6104776707604625bd1733edfd98548d52408c WatchSource:0}: Error finding container 4b5f19ac92ae5427c430b52ffb6104776707604625bd1733edfd98548d52408c: Status 404 returned error can't find the container with id 4b5f19ac92ae5427c430b52ffb6104776707604625bd1733edfd98548d52408c Nov 04 00:58:04.588516 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:04.588480 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c9c4bcdc-q9hwj" event={"ID":"53df385f-18fa-4ca3-a39a-0d5c0f4bc539","Type":"ContainerStarted","Data":"4b5f19ac92ae5427c430b52ffb6104776707604625bd1733edfd98548d52408c"} Nov 04 00:58:08.605573 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:08.605540 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c9c4bcdc-q9hwj" event={"ID":"53df385f-18fa-4ca3-a39a-0d5c0f4bc539","Type":"ContainerStarted","Data":"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e"} Nov 04 00:58:08.763870 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:08.763820 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74c9c4bcdc-q9hwj" podStartSLOduration=1.873538324 podStartE2EDuration="5.763807264s" totalImagesPullingTime="3.89026894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:08.716661329 +0000 UTC m=+57.475323503" watchObservedRunningTime="2026-11-04 00:58:08.763807264 +0000 UTC m=+57.522469424" Nov 04 00:58:13.468333 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:13.468292 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:13.468737 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:13.468376 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:13.472855 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:13.472833 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:13.624257 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:58:13.624228 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:58:13.828458 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:58:13.828427 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 00:58:43.830068 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:58:43.830027 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 00:59:06.799192 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:06.799107 4486 generic.go:356] "Generic (PLEG): container finished" podID="6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" containerID="fcabbe1f88e09616f964bdeea1ccddd6207c07c252120842928d51bb7068b228" exitCode=0 Nov 04 00:59:06.799635 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:06.799182 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" event={"ID":"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1","Type":"ContainerDied","Data":"fcabbe1f88e09616f964bdeea1ccddd6207c07c252120842928d51bb7068b228"} Nov 04 00:59:07.941772 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:07.941748 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:59:08.019148 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019126 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\") pod \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " Nov 04 00:59:08.019278 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019159 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\") pod \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " Nov 04 00:59:08.019278 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019183 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\") pod \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " Nov 04 00:59:08.019278 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019214 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\" (UniqueName: \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\") pod \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\" (UID: \"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1\") " Nov 04 00:59:08.019483 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019458 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path" pod "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" (UID: "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:59:08.019549 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019475 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle" pod "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" (UID: "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:08.019549 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.019512 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle" pod "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" (UID: "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:08.021327 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.021308 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6" pod "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" (UID: "6b8608fb-515b-42c3-ae4f-6eaf612c2ae1"). InnerVolumeSpecName "kube-api-access-z79h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:08.119944 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.119906 4486 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-archives-path\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:08.119944 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.119942 4486 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-trusted-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:08.119944 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.119952 4486 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-service-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:08.120167 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.119961 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79h6\" (UniqueName: \"kubernetes.io/projected/6b8608fb-515b-42c3-ae4f-6eaf612c2ae1-kube-api-access-z79h6\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:08.807176 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.807140 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" event={"ID":"6b8608fb-515b-42c3-ae4f-6eaf612c2ae1","Type":"ContainerDied","Data":"bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b"} Nov 04 00:59:08.807176 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.807160 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-dqjq2-k5dll" Nov 04 00:59:08.807376 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:08.807193 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0f84aa3049aa2e73e88d4f19d7f0cf184fed170bb3493c29f0dd56fc3bc24b" Nov 04 00:59:13.828837 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:59:13.828799 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 00:59:18.581930 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.581896 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 00:59:18.582405 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.582171 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" containerName="insights-gathering" Nov 04 00:59:18.582405 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.582183 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" containerName="insights-gathering" Nov 04 00:59:18.582405 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.582242 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b8608fb-515b-42c3-ae4f-6eaf612c2ae1" containerName="insights-gathering" Nov 04 00:59:18.585165 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.585140 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.639080 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.639059 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 00:59:18.696193 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696158 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696328 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696263 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696328 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696308 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696434 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696410 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696488 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696468 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696560 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696539 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqs4v\" (UniqueName: \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696603 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696590 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.696705 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.696688 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797244 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797323 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797726 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqs4v\" (UniqueName: \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797803 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797873 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797907 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.797945 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.798007 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.798096 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.798223 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.798592 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.798915 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.801881 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.802726 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.802034 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.803512 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.803328 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.858024 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.857948 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqs4v\" (UniqueName: \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\") pod \"console-7c6cf796dd-xdb2m\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:18.895856 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:18.895832 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:19.264099 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:19.264031 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 00:59:19.266885 ip-10-0-129-114 kubenswrapper[4486]: W1104 00:59:19.266856 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadbb705_b01c_4a0e_a9b1_eb1f6e971e4a.slice/crio-bb7bd3ffda1fa917f3240780291a22f24668a9d7e15b686e0565bf381bbb8b73 WatchSource:0}: Error finding container bb7bd3ffda1fa917f3240780291a22f24668a9d7e15b686e0565bf381bbb8b73: Status 404 returned error can't find the container with id bb7bd3ffda1fa917f3240780291a22f24668a9d7e15b686e0565bf381bbb8b73 Nov 04 00:59:19.844178 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:19.844132 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cf796dd-xdb2m" event={"ID":"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a","Type":"ContainerStarted","Data":"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e"} Nov 04 00:59:19.844543 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:19.844198 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cf796dd-xdb2m" event={"ID":"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a","Type":"ContainerStarted","Data":"bb7bd3ffda1fa917f3240780291a22f24668a9d7e15b686e0565bf381bbb8b73"} Nov 04 00:59:19.954225 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:19.954179 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c6cf796dd-xdb2m" podStartSLOduration=1.954165124 podStartE2EDuration="1.954165124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:59:19.953833646 +0000 UTC m=+128.712495808" watchObservedRunningTime="2026-11-04 00:59:19.954165124 +0000 UTC m=+128.712827285" Nov 04 00:59:28.896608 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:28.896520 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:28.896608 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:28.896594 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:28.902072 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:28.902037 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:29.880834 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:29.880810 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 00:59:30.038092 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:30.038059 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:59:43.830238 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:59:43.830197 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 00:59:55.057222 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.057162 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-74c9c4bcdc-q9hwj" podUID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" containerName="console" containerID="cri-o://59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e" gracePeriod=15 Nov 04 00:59:55.309745 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.309690 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:59:55.443808 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443781 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.443939 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443820 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\" (UniqueName: \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.443939 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443857 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.443939 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443879 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.443939 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443902 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\" (UniqueName: \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.444253 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.443968 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.444253 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444045 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.444253 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444084 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\") pod \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\" (UID: \"53df385f-18fa-4ca3-a39a-0d5c0f4bc539\") " Nov 04 00:59:55.444423 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444255 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:59:55.444423 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444275 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:55.444532 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444427 4486 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-oauth-serving-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.444532 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444447 4486 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-tmp\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.444532 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444430 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:55.444684 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444526 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:55.444684 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.444553 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:55.446191 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.446168 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:55.446312 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.446284 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "kube-api-access-jrz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:55.446357 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.446324 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert" pod "53df385f-18fa-4ca3-a39a-0d5c0f4bc539" (UID: "53df385f-18fa-4ca3-a39a-0d5c0f4bc539"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:59:55.544965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.544945 4486 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-trusted-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.544965 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.544966 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrz82\" (UniqueName: \"kubernetes.io/projected/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-kube-api-access-jrz82\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.545110 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.544977 4486 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-serving-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.545110 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.545009 4486 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-config\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.545110 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.545019 4486 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.545110 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.545027 4486 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53df385f-18fa-4ca3-a39a-0d5c0f4bc539-console-oauth-config\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:55.967346 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.967310 4486 generic.go:356] "Generic (PLEG): container finished" podID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" containerID="59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e" exitCode=0 Nov 04 00:59:55.967497 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.967376 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c9c4bcdc-q9hwj" Nov 04 00:59:55.967497 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.967381 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c9c4bcdc-q9hwj" event={"ID":"53df385f-18fa-4ca3-a39a-0d5c0f4bc539","Type":"ContainerDied","Data":"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e"} Nov 04 00:59:55.967497 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.967460 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c9c4bcdc-q9hwj" event={"ID":"53df385f-18fa-4ca3-a39a-0d5c0f4bc539","Type":"ContainerDied","Data":"4b5f19ac92ae5427c430b52ffb6104776707604625bd1733edfd98548d52408c"} Nov 04 00:59:55.967602 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.967488 4486 scope.go:122] "RemoveContainer" containerID="59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e" Nov 04 00:59:55.976119 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.976094 4486 scope.go:122] "RemoveContainer" containerID="59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e" Nov 04 00:59:55.976384 ip-10-0-129-114 kubenswrapper[4486]: E1104 00:59:55.976367 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e\": container with ID starting with 59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e not found: ID does not exist" containerID="59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e" Nov 04 00:59:55.976450 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:55.976392 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e"} err="failed to get container status \"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e\": rpc error: code = NotFound desc = could not find container \"59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e\": container with ID starting with 59474e047dcb9625b868892ec40fde206bce583a0df6580e366d4b34865d632e not found: ID does not exist" Nov 04 00:59:56.122132 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:56.122103 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:59:56.169325 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:56.169301 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c9c4bcdc-q9hwj"] Nov 04 00:59:57.345530 ip-10-0-129-114 kubenswrapper[4486]: I1104 00:59:57.345498 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" path="/var/lib/kubelet/pods/53df385f-18fa-4ca3-a39a-0d5c0f4bc539/volumes" Nov 04 01:00:13.829069 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:00:13.829027 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:00:37.773642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.773607 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77969d4f4-k4j44"] Nov 04 01:00:37.774148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.773888 4486 cpu_manager.go:414] "containerMap: removing container" podUID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" containerName="console" Nov 04 01:00:37.774148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.773898 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" containerName="console" Nov 04 01:00:37.774148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.773951 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="53df385f-18fa-4ca3-a39a-0d5c0f4bc539" containerName="console" Nov 04 01:00:37.776651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.776635 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.837856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.837828 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-service-ca\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.837973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.837862 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-oauth-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.837973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.837878 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-console-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.837973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.837896 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-trusted-ca-bundle\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.838113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.838046 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-oauth-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.838113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.838085 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.838113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.838101 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmh4\" (UniqueName: \"kubernetes.io/projected/95005736-3a77-4e77-aced-5b545a149eb8-kube-api-access-vtmh4\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.838242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.838152 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95005736-3a77-4e77-aced-5b545a149eb8-tmp\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.874288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.874266 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77969d4f4-k4j44"] Nov 04 01:00:37.938705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938679 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-service-ca\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938720 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-oauth-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938740 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-console-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938797 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-trusted-ca-bundle\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938848 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-oauth-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938891 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938912 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmh4\" (UniqueName: \"kubernetes.io/projected/95005736-3a77-4e77-aced-5b545a149eb8-kube-api-access-vtmh4\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.938966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.938941 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95005736-3a77-4e77-aced-5b545a149eb8-tmp\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.939497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.939477 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95005736-3a77-4e77-aced-5b545a149eb8-tmp\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.939645 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.939499 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-console-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.939746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.939725 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-service-ca\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.939894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.939871 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-oauth-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.939935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.939918 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95005736-3a77-4e77-aced-5b545a149eb8-trusted-ca-bundle\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.941267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.941247 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-oauth-config\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.941479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.941461 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95005736-3a77-4e77-aced-5b545a149eb8-console-serving-cert\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:37.994963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:37.994935 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmh4\" (UniqueName: \"kubernetes.io/projected/95005736-3a77-4e77-aced-5b545a149eb8-kube-api-access-vtmh4\") pod \"console-77969d4f4-k4j44\" (UID: \"95005736-3a77-4e77-aced-5b545a149eb8\") " pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:38.087024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:38.086971 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:38.448643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:38.448559 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77969d4f4-k4j44"] Nov 04 01:00:38.451277 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:00:38.451245 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95005736_3a77_4e77_aced_5b545a149eb8.slice/crio-f9e12b546bf632d4939ab3ea3054e02281bd0dfb53e8cfeb2a8b19607b9e4196 WatchSource:0}: Error finding container f9e12b546bf632d4939ab3ea3054e02281bd0dfb53e8cfeb2a8b19607b9e4196: Status 404 returned error can't find the container with id f9e12b546bf632d4939ab3ea3054e02281bd0dfb53e8cfeb2a8b19607b9e4196 Nov 04 01:00:39.109460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:39.109418 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77969d4f4-k4j44" event={"ID":"95005736-3a77-4e77-aced-5b545a149eb8","Type":"ContainerStarted","Data":"edb496b0ff48d47eb562727fc167f138c24596d5b688a26ccdb319b0dfa45a96"} Nov 04 01:00:39.109843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:39.109479 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77969d4f4-k4j44" event={"ID":"95005736-3a77-4e77-aced-5b545a149eb8","Type":"ContainerStarted","Data":"f9e12b546bf632d4939ab3ea3054e02281bd0dfb53e8cfeb2a8b19607b9e4196"} Nov 04 01:00:39.265540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:39.265485 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77969d4f4-k4j44" podStartSLOduration=2.265471545 podStartE2EDuration="2.265471545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:00:39.218314087 +0000 UTC m=+207.976976252" watchObservedRunningTime="2026-11-04 01:00:39.265471545 +0000 UTC m=+208.024133684" Nov 04 01:00:43.828483 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:00:43.828442 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:00:48.087877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:48.087837 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:48.088310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:48.087898 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:48.092396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:48.092372 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:48.153706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:48.153685 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77969d4f4-k4j44" Nov 04 01:00:48.336791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:00:48.336759 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 01:01:13.356236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.356179 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c6cf796dd-xdb2m" podUID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" containerName="console" containerID="cri-o://97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e" gracePeriod=15 Nov 04 01:01:13.587089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.587064 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 01:01:13.701813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701741 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.701813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701786 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.701813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701805 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701844 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701872 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701903 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\" (UniqueName: \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.701967 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.702018 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\") pod \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\" (UID: \"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a\") " Nov 04 01:01:13.702480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.702359 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:01:13.702480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.702367 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:01:13.702596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.702426 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:01:13.702687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.702662 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:01:13.703124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.703093 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:01:13.704263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.704236 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:01:13.704372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.704267 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:01:13.704437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.704413 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v" pod "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" (UID: "dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a"). InnerVolumeSpecName "kube-api-access-hqs4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:01:13.802787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802761 4486 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-config\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802784 4486 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802795 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqs4v\" (UniqueName: \"kubernetes.io/projected/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-kube-api-access-hqs4v\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802807 4486 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-tmp\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802815 4486 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-serving-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802824 4486 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-oauth-serving-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802834 4486 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-console-oauth-config\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.802934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:13.802842 4486 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a-trusted-ca-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:01:13.828179 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:01:13.828149 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:01:14.234714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.234683 4486 generic.go:356] "Generic (PLEG): container finished" podID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" containerID="97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e" exitCode=0 Nov 04 01:01:14.234895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.234732 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6cf796dd-xdb2m" Nov 04 01:01:14.234895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.234733 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cf796dd-xdb2m" event={"ID":"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a","Type":"ContainerDied","Data":"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e"} Nov 04 01:01:14.234895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.234871 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6cf796dd-xdb2m" event={"ID":"dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a","Type":"ContainerDied","Data":"bb7bd3ffda1fa917f3240780291a22f24668a9d7e15b686e0565bf381bbb8b73"} Nov 04 01:01:14.235069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.234901 4486 scope.go:122] "RemoveContainer" containerID="97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e" Nov 04 01:01:14.243871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.243856 4486 scope.go:122] "RemoveContainer" containerID="97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e" Nov 04 01:01:14.244151 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:01:14.244135 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e\": container with ID starting with 97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e not found: ID does not exist" containerID="97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e" Nov 04 01:01:14.244208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.244158 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e"} err="failed to get container status \"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e\": rpc error: code = NotFound desc = could not find container \"97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e\": container with ID starting with 97b934f1dbda3ba5abe3727f54e4f4fc1b07b65a90048ab7e18166eb2f38288e not found: ID does not exist" Nov 04 01:01:14.389666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.389635 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 01:01:14.437029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:14.437000 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6cf796dd-xdb2m"] Nov 04 01:01:15.345226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:01:15.345194 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" path="/var/lib/kubelet/pods/dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a/volumes" Nov 04 01:01:43.828961 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:01:43.828921 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:02:11.353141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:02:11.353116 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:02:11.355863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:02:11.355844 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:02:13.829508 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:02:13.829470 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:02:43.829708 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:02:43.829668 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:03:13.426551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.426517 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw"] Nov 04 01:03:13.426948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.426777 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" containerName="console" Nov 04 01:03:13.426948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.426788 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" containerName="console" Nov 04 01:03:13.426948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.426847 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadbb705-b01c-4a0e-a9b1-eb1f6e971e4a" containerName="console" Nov 04 01:03:13.429629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.429606 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.482689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.482665 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9lnzk\"" Nov 04 01:03:13.492022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.492002 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 04 01:03:13.492208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.492192 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 04 01:03:13.529291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.529268 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.529368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.529304 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.529368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.529331 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcxx\" (UniqueName: \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.539073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.539047 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw"] Nov 04 01:03:13.548784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.548760 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85"] Nov 04 01:03:13.552415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.552399 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.622130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.622104 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85"] Nov 04 01:03:13.630571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630549 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4qq\" (UniqueName: \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.630659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630590 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.630659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630624 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.630728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630686 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.630815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630792 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcxx\" (UniqueName: \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.630850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.630839 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.631032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.631015 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.631082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.631043 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.644867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.644841 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-b48lx\"" Nov 04 01:03:13.694494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.694439 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcxx\" (UniqueName: \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.731492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.731463 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.731608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.731499 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4qq\" (UniqueName: \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.731608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.731525 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.731826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.731811 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.731887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.731868 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.744434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.744416 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:13.790877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.790844 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4qq\" (UniqueName: \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:13.830877 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:03:13.830795 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:03:13.867425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:13.867396 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:14.105016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:14.104972 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw"] Nov 04 01:03:14.108008 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:14.107962 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b84c42_f634_4a83_9a71_62caf03a995d.slice/crio-a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43 WatchSource:0}: Error finding container a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43: Status 404 returned error can't find the container with id a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43 Nov 04 01:03:14.110056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:14.110035 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:03:14.230540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:14.230516 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85"] Nov 04 01:03:14.233041 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:14.233010 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29da81f9_6162_4fb5_acd5_06eb5fe72c32.slice/crio-114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3 WatchSource:0}: Error finding container 114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3: Status 404 returned error can't find the container with id 114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3 Nov 04 01:03:14.626185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:14.626150 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerStarted","Data":"114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3"} Nov 04 01:03:14.627273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:14.627243 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" event={"ID":"65b84c42-f634-4a83-9a71-62caf03a995d","Type":"ContainerStarted","Data":"a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43"} Nov 04 01:03:15.676784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.676747 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs"] Nov 04 01:03:15.680333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.680308 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.746762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.746730 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.747092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.746777 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.747092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.746806 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44lj\" (UniqueName: \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.777941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.777914 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-572cc\"" Nov 04 01:03:15.778754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.778731 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs"] Nov 04 01:03:15.847487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.847458 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.847610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.847497 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.847610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.847528 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44lj\" (UniqueName: \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.848104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.848071 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.848377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.848358 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:15.904025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:15.904001 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44lj\" (UniqueName: \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:16.000695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:16.000599 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:16.386354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:16.386322 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs"] Nov 04 01:03:16.420610 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:16.420563 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbf1c56_0a2f_4e1d_b6f4_75b4980162a8.slice/crio-a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779 WatchSource:0}: Error finding container a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779: Status 404 returned error can't find the container with id a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779 Nov 04 01:03:16.636957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:16.636875 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" event={"ID":"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8","Type":"ContainerStarted","Data":"a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779"} Nov 04 01:03:17.455430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.455401 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd"] Nov 04 01:03:17.458605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.458580 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.508467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.508439 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r75km\"" Nov 04 01:03:17.513544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.513511 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd"] Nov 04 01:03:17.547849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.547820 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v"] Nov 04 01:03:17.551587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.551559 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.563328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.563301 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.563448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.563362 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.563448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.563428 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdf2\" (UniqueName: \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.607336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.607311 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v"] Nov 04 01:03:17.609401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.609379 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-c86rp\"" Nov 04 01:03:17.651673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.651644 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t"] Nov 04 01:03:17.655055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.655026 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.665588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665553 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.665703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665627 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfrh\" (UniqueName: \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.665703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665670 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.665805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665727 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.665805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665756 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdf2\" (UniqueName: \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.665805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.665785 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.666336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.666314 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.666658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.666639 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.709350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.709266 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mnhlm\"" Nov 04 01:03:17.710626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.710599 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t"] Nov 04 01:03:17.721997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.721956 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdf2\" (UniqueName: \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.746482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.746458 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls"] Nov 04 01:03:17.750087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.750066 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.766405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.766524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766431 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.766524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766506 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfrh\" (UniqueName: \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.766644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766554 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98qj\" (UniqueName: \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.766644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766599 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.766644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766621 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.766911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.766893 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.767071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.767041 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.775235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.775213 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:17.810549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.810523 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-v78zb\"" Nov 04 01:03:17.814199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.814172 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls"] Nov 04 01:03:17.829648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.829616 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfrh\" (UniqueName: \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.867347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867316 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.867472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867428 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.867536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867476 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.867536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867530 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98qj\" (UniqueName: \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.867643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867565 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdwx\" (UniqueName: \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.867643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867610 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.867859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867834 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.868015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.867955 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.869663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.869641 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:17.922571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.922541 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98qj\" (UniqueName: \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:17.968732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.968513 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.968732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.968587 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdwx\" (UniqueName: \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.968732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.968667 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.969178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.969149 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.969354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.969328 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:17.974439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:17.974418 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:18.025095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.025064 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdwx\" (UniqueName: \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:18.069536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.069508 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:18.161701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.159642 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd"] Nov 04 01:03:18.167423 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:18.167386 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode691e884_950b_43b9_8f7c_7748cc9056ae.slice/crio-cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce WatchSource:0}: Error finding container cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce: Status 404 returned error can't find the container with id cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce Nov 04 01:03:18.258710 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.258685 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v"] Nov 04 01:03:18.260660 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:18.260621 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4523d68a_7f7b_4010_a621_c99e5d7bf9be.slice/crio-5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528 WatchSource:0}: Error finding container 5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528: Status 404 returned error can't find the container with id 5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528 Nov 04 01:03:18.367256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.367225 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t"] Nov 04 01:03:18.370044 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:18.370012 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ef04dd_8c47_47b1_b965_33cab1207136.slice/crio-e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792 WatchSource:0}: Error finding container e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792: Status 404 returned error can't find the container with id e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792 Nov 04 01:03:18.449169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.449138 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls"] Nov 04 01:03:18.455482 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:18.455449 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafc78b1_6d9b_4717_bd7e_6d7b0ab00937.slice/crio-f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c WatchSource:0}: Error finding container f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c: Status 404 returned error can't find the container with id f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c Nov 04 01:03:18.647398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.647362 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerStarted","Data":"5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528"} Nov 04 01:03:18.648703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.648677 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" event={"ID":"e691e884-950b-43b9-8f7c-7748cc9056ae","Type":"ContainerStarted","Data":"cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce"} Nov 04 01:03:18.650078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.650023 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" event={"ID":"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937","Type":"ContainerStarted","Data":"f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c"} Nov 04 01:03:18.651287 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:18.651270 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" event={"ID":"81ef04dd-8c47-47b1-b965-33cab1207136","Type":"ContainerStarted","Data":"e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792"} Nov 04 01:03:19.685479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.685436 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd"] Nov 04 01:03:19.689862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.689484 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.784498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.784437 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.784662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.784519 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.784662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.784593 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqrc\" (UniqueName: \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.790206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.790175 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd"] Nov 04 01:03:19.790344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.790250 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw"] Nov 04 01:03:19.796083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.796062 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.860213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.859904 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw"] Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885515 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885622 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqrc\" (UniqueName: \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885669 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsp4v\" (UniqueName: \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885703 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885772 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.885798 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.886248 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.886929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.886883 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.946560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.946445 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqrc\" (UniqueName: \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:19.988138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.987043 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.988138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.987128 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsp4v\" (UniqueName: \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.988138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.987164 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.988138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.987577 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:19.988138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:19.987812 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:20.009977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.009742 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:20.055928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.055870 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsp4v\" (UniqueName: \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:20.123324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.119132 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:20.445961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.445926 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd"] Nov 04 01:03:20.453399 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:20.453366 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28b03d5_a3b9_4087_8eda_fe37a9c79bf1.slice/crio-3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad WatchSource:0}: Error finding container 3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad: Status 404 returned error can't find the container with id 3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad Nov 04 01:03:20.559119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.559063 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw"] Nov 04 01:03:20.565598 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:20.565567 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113c9921_1e9f_42ee_a4d5_ee5f27e3fb64.slice/crio-1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878 WatchSource:0}: Error finding container 1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878: Status 404 returned error can't find the container with id 1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878 Nov 04 01:03:20.670847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.670763 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerStarted","Data":"3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad"} Nov 04 01:03:20.673942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:20.673094 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerStarted","Data":"1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878"} Nov 04 01:03:21.286617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.286579 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:03:21.293173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.292265 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.347160 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.347132 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:03:21.347385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.347337 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 01:03:21.366115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.366088 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 01:03:21.398400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.398367 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:03:21.398576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.398558 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwrps\"" Nov 04 01:03:21.408110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408083 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408141 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zb8\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408175 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408213 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408291 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408366 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408427 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408461 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.408611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.408531 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.509610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509565 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.509772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509621 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.509772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509658 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.509772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509689 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.509772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509741 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.510028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509779 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.510028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509804 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zb8\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.510028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509834 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.510028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.509876 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.512699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.510761 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.512699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.511027 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.512699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.512157 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.512699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.512498 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.512699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.512522 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.524967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.524095 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.533504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.533473 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.571768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.571739 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.574292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.574263 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zb8\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\") pod \"maas-default-gateway-openshift-default-694db55c87-q6c5l\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.578030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.576969 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv"] Nov 04 01:03:21.583090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.582695 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.609724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.609687 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:21.633466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.633206 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-8dfxd\"" Nov 04 01:03:21.638163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.638138 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv"] Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715727 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwbg\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-kube-api-access-gmwbg\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715779 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715812 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715842 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715874 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715902 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.715971 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6536c33b-bd9a-4f7b-b963-985785fa85ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.716033 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.716255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.716062 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817357 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwbg\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-kube-api-access-gmwbg\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817413 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817447 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817477 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817508 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817537 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817599 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6536c33b-bd9a-4f7b-b963-985785fa85ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.817650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817646 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.818055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.817672 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.818627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.818122 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.818627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.818489 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.820194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.819226 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.820194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.819826 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6536c33b-bd9a-4f7b-b963-985785fa85ff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.820194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.820092 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.826119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.826059 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.835147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.835110 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.887372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.887278 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwbg\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-kube-api-access-gmwbg\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:21.903523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:21.903475 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6536c33b-bd9a-4f7b-b963-985785fa85ff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv\" (UID: \"6536c33b-bd9a-4f7b-b963-985785fa85ff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:22.068895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:22.067606 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:03:22.077194 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:03:22.077136 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c18ddb_ccc6_403d_b712_ba202a506ea3.slice/crio-2a578209bae5ae0eabbf2a784e2073962bfc7373146831e17e9cdd9b9c6bae05 WatchSource:0}: Error finding container 2a578209bae5ae0eabbf2a784e2073962bfc7373146831e17e9cdd9b9c6bae05: Status 404 returned error can't find the container with id 2a578209bae5ae0eabbf2a784e2073962bfc7373146831e17e9cdd9b9c6bae05 Nov 04 01:03:22.202235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:22.202196 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:22.685220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:22.685148 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv"] Nov 04 01:03:22.717561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:22.717530 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" event={"ID":"6536c33b-bd9a-4f7b-b963-985785fa85ff","Type":"ContainerStarted","Data":"09f31575bdd092dae67410bf5c4f1f1299a8d81acb138f2deb805ae46cb174e9"} Nov 04 01:03:22.728935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:22.728864 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerStarted","Data":"2a578209bae5ae0eabbf2a784e2073962bfc7373146831e17e9cdd9b9c6bae05"} Nov 04 01:03:28.815699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:28.815663 4486 generic.go:356] "Generic (PLEG): container finished" podID="65b84c42-f634-4a83-9a71-62caf03a995d" containerID="8c944cd63d1cb140b5df93e68b23ee7d7bfc2fdeb9342cbf96c90326cdb7e457" exitCode=0 Nov 04 01:03:28.816290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:28.815759 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" event={"ID":"65b84c42-f634-4a83-9a71-62caf03a995d","Type":"ContainerDied","Data":"8c944cd63d1cb140b5df93e68b23ee7d7bfc2fdeb9342cbf96c90326cdb7e457"} Nov 04 01:03:29.826949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:29.826900 4486 generic.go:356] "Generic (PLEG): container finished" podID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerID="f41f0f56a3e9e840584390d196b652bece006d3512cb3b71ed20350b2f2a81b5" exitCode=0 Nov 04 01:03:29.827428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:29.827017 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerDied","Data":"f41f0f56a3e9e840584390d196b652bece006d3512cb3b71ed20350b2f2a81b5"} Nov 04 01:03:30.836498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.835556 4486 generic.go:356] "Generic (PLEG): container finished" podID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerID="507986f4a9417fd1ee5a2b522cd89f9ba7cdf5e2d02b080175b15d2c9d71c17e" exitCode=0 Nov 04 01:03:30.836498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.835654 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerDied","Data":"507986f4a9417fd1ee5a2b522cd89f9ba7cdf5e2d02b080175b15d2c9d71c17e"} Nov 04 01:03:30.840637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.839960 4486 generic.go:356] "Generic (PLEG): container finished" podID="65b84c42-f634-4a83-9a71-62caf03a995d" containerID="6c2be312188beae7e67bcb3a46383622b8aec98e4eb12d657bf6366082aa2b1b" exitCode=0 Nov 04 01:03:30.840637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.840087 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" event={"ID":"65b84c42-f634-4a83-9a71-62caf03a995d","Type":"ContainerDied","Data":"6c2be312188beae7e67bcb3a46383622b8aec98e4eb12d657bf6366082aa2b1b"} Nov 04 01:03:30.846017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.845767 4486 generic.go:356] "Generic (PLEG): container finished" podID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerID="186e345c0daea0a9c634e914faf7e344af827ce83af4eb145c1177314fb912da" exitCode=0 Nov 04 01:03:30.846017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.845861 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" event={"ID":"e691e884-950b-43b9-8f7c-7748cc9056ae","Type":"ContainerDied","Data":"186e345c0daea0a9c634e914faf7e344af827ce83af4eb145c1177314fb912da"} Nov 04 01:03:30.849477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.849454 4486 generic.go:356] "Generic (PLEG): container finished" podID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerID="12ccd330ef9663ae5ec1f29dae603db8adda388f5e82c090b4842669cb3d6d44" exitCode=0 Nov 04 01:03:30.849556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:30.849495 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" event={"ID":"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937","Type":"ContainerDied","Data":"12ccd330ef9663ae5ec1f29dae603db8adda388f5e82c090b4842669cb3d6d44"} Nov 04 01:03:31.640386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.640356 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.640517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.640465 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.640580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.640513 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.789384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.788877 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.789384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.788971 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.789384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.789040 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:03:31.858669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.857715 4486 generic.go:356] "Generic (PLEG): container finished" podID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerID="baad0e721ff44cf1bfc9624eb7d5de1dc8fc2c88442f46e229dd5f274085641a" exitCode=0 Nov 04 01:03:31.858669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.857820 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerDied","Data":"baad0e721ff44cf1bfc9624eb7d5de1dc8fc2c88442f46e229dd5f274085641a"} Nov 04 01:03:31.861335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.860709 4486 generic.go:356] "Generic (PLEG): container finished" podID="81ef04dd-8c47-47b1-b965-33cab1207136" containerID="391e1bf741d40b3a52a2233ea9a26019965f6dca62239cf0c53b2bf6427eb3f2" exitCode=0 Nov 04 01:03:31.861335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.860793 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" event={"ID":"81ef04dd-8c47-47b1-b965-33cab1207136","Type":"ContainerDied","Data":"391e1bf741d40b3a52a2233ea9a26019965f6dca62239cf0c53b2bf6427eb3f2"} Nov 04 01:03:31.862769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:31.862746 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerStarted","Data":"d7406a3eb9a46d23c655a0b9e1bab43429ed5d377d16b933795ff3b86104ee1b"} Nov 04 01:03:32.089062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.089012 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podStartSLOduration=1.531992454 podStartE2EDuration="11.088994876s" totalImagesPullingTime="9.557002422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:32.087294214 +0000 UTC m=+380.845956390" watchObservedRunningTime="2026-11-04 01:03:32.088994876 +0000 UTC m=+380.847657033" Nov 04 01:03:32.611098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.611055 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:32.617326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.617298 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:32.870785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.870754 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" event={"ID":"6536c33b-bd9a-4f7b-b963-985785fa85ff","Type":"ContainerStarted","Data":"77786edc2fbfccd5fcfc24d39c74615aebe5a37b8d05136720f57a593a112b35"} Nov 04 01:03:32.872586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.872563 4486 generic.go:356] "Generic (PLEG): container finished" podID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerID="f8ca092d963e0f2cbf947fe374a39719d50e347f64319c87b20bb1610735f6b8" exitCode=0 Nov 04 01:03:32.872697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.872628 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" event={"ID":"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937","Type":"ContainerDied","Data":"f8ca092d963e0f2cbf947fe374a39719d50e347f64319c87b20bb1610735f6b8"} Nov 04 01:03:32.876033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.876013 4486 generic.go:356] "Generic (PLEG): container finished" podID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerID="acebf1f4cd4ba2b72f4d96aa7011cc50721a7a0440cdd8155bf721aa57cdccd6" exitCode=0 Nov 04 01:03:32.876145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.876042 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerDied","Data":"acebf1f4cd4ba2b72f4d96aa7011cc50721a7a0440cdd8155bf721aa57cdccd6"} Nov 04 01:03:32.879146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.879123 4486 generic.go:356] "Generic (PLEG): container finished" podID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerID="cfdf9c14341d38e9652aafa7ed309e1512745f07c207f535c3ba0e9aea27e8bd" exitCode=0 Nov 04 01:03:32.880071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.880044 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" event={"ID":"e691e884-950b-43b9-8f7c-7748cc9056ae","Type":"ContainerDied","Data":"cfdf9c14341d38e9652aafa7ed309e1512745f07c207f535c3ba0e9aea27e8bd"} Nov 04 01:03:32.880174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.880109 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:32.881099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.881083 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:03:32.986769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:32.986720 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" podStartSLOduration=2.904461085 podStartE2EDuration="11.986704707s" totalImagesPullingTime="9.082243622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:32.985495935 +0000 UTC m=+381.744158096" watchObservedRunningTime="2026-11-04 01:03:32.986704707 +0000 UTC m=+381.745366868" Nov 04 01:03:33.203448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.203377 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:33.209089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.209059 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:33.886924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.886889 4486 generic.go:356] "Generic (PLEG): container finished" podID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerID="5a9390f0416d9a60a97f0aa7b076b9d08a5530a1c0a0b01a50bd3799727b93df" exitCode=0 Nov 04 01:03:33.887482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.887006 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerDied","Data":"5a9390f0416d9a60a97f0aa7b076b9d08a5530a1c0a0b01a50bd3799727b93df"} Nov 04 01:03:33.889629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.889597 4486 generic.go:356] "Generic (PLEG): container finished" podID="81ef04dd-8c47-47b1-b965-33cab1207136" containerID="d0ba3831ad819cdbca2292d21a9d5443cd9dbe541149dc499df1a37fd3e3f607" exitCode=0 Nov 04 01:03:33.889747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.889721 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" event={"ID":"81ef04dd-8c47-47b1-b965-33cab1207136","Type":"ContainerDied","Data":"d0ba3831ad819cdbca2292d21a9d5443cd9dbe541149dc499df1a37fd3e3f607"} Nov 04 01:03:33.892127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.892086 4486 generic.go:356] "Generic (PLEG): container finished" podID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerID="f687c9efbdf42396df2b74b2c4caca061daeb4634b06cf4d98807e0afc306249" exitCode=0 Nov 04 01:03:33.892205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.892169 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerDied","Data":"f687c9efbdf42396df2b74b2c4caca061daeb4634b06cf4d98807e0afc306249"} Nov 04 01:03:33.893841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.893820 4486 generic.go:356] "Generic (PLEG): container finished" podID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerID="4c893dc31dcab5e6ade0419f13b4ddb964586aecfb618edd19c1872a25b64769" exitCode=0 Nov 04 01:03:33.893921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.893906 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" event={"ID":"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8","Type":"ContainerDied","Data":"4c893dc31dcab5e6ade0419f13b4ddb964586aecfb618edd19c1872a25b64769"} Nov 04 01:03:33.894305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.894286 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:33.895166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:33.895149 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv" Nov 04 01:03:34.904968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:34.904213 4486 generic.go:356] "Generic (PLEG): container finished" podID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerID="9579fa69c6036e5a70e57ab8ed23950aedcfec0d799a48f5541979ddc5276482" exitCode=0 Nov 04 01:03:34.904968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:34.904595 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerDied","Data":"9579fa69c6036e5a70e57ab8ed23950aedcfec0d799a48f5541979ddc5276482"} Nov 04 01:03:34.916007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:34.914604 4486 generic.go:356] "Generic (PLEG): container finished" podID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerID="58ad278ec8f88f114acb8fea6884d8653e6a0ed6254874d081da44e067dc23ea" exitCode=0 Nov 04 01:03:34.916007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:34.915728 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerDied","Data":"58ad278ec8f88f114acb8fea6884d8653e6a0ed6254874d081da44e067dc23ea"} Nov 04 01:03:41.991479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:41.990430 4486 generic.go:356] "Generic (PLEG): container finished" podID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerID="878aa669731eea5aedf748931cef9df7c83d39917cab39ced1ac85d8df92fcb8" exitCode=0 Nov 04 01:03:41.991479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:41.990481 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" event={"ID":"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8","Type":"ContainerDied","Data":"878aa669731eea5aedf748931cef9df7c83d39917cab39ced1ac85d8df92fcb8"} Nov 04 01:03:43.867063 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:03:43.867021 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:03:49.072286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:49.072246 4486 generic.go:356] "Generic (PLEG): container finished" podID="65b84c42-f634-4a83-9a71-62caf03a995d" containerID="e67029659773fd9c60b1c1edfba7f08d1861b094f19aa5c6f24479d52ff07813" exitCode=0 Nov 04 01:03:49.072744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:49.072309 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" event={"ID":"65b84c42-f634-4a83-9a71-62caf03a995d","Type":"ContainerDied","Data":"e67029659773fd9c60b1c1edfba7f08d1861b094f19aa5c6f24479d52ff07813"} Nov 04 01:03:50.290719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.290687 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:50.330209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.330154 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\" (UniqueName: \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\") pod \"65b84c42-f634-4a83-9a71-62caf03a995d\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " Nov 04 01:03:50.330319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.330228 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\") pod \"65b84c42-f634-4a83-9a71-62caf03a995d\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " Nov 04 01:03:50.330319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.330279 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\") pod \"65b84c42-f634-4a83-9a71-62caf03a995d\" (UID: \"65b84c42-f634-4a83-9a71-62caf03a995d\") " Nov 04 01:03:50.332370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.332221 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle" pod "65b84c42-f634-4a83-9a71-62caf03a995d" (UID: "65b84c42-f634-4a83-9a71-62caf03a995d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:50.334806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.334778 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx" pod "65b84c42-f634-4a83-9a71-62caf03a995d" (UID: "65b84c42-f634-4a83-9a71-62caf03a995d"). InnerVolumeSpecName "kube-api-access-gtcxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:50.338249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.338209 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util" pod "65b84c42-f634-4a83-9a71-62caf03a995d" (UID: "65b84c42-f634-4a83-9a71-62caf03a995d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:50.432582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.432553 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtcxx\" (UniqueName: \"kubernetes.io/projected/65b84c42-f634-4a83-9a71-62caf03a995d-kube-api-access-gtcxx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:50.432677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.432589 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:50.432677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:50.432606 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65b84c42-f634-4a83-9a71-62caf03a995d-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:51.092065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.092028 4486 generic.go:356] "Generic (PLEG): container finished" podID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerID="814458671b9528470e97fc592f69f74b816706bfabe6a43eb67435bd7c5a4351" exitCode=0 Nov 04 01:03:51.092260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.092167 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" event={"ID":"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937","Type":"ContainerDied","Data":"814458671b9528470e97fc592f69f74b816706bfabe6a43eb67435bd7c5a4351"} Nov 04 01:03:51.096417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.096389 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" event={"ID":"65b84c42-f634-4a83-9a71-62caf03a995d","Type":"ContainerDied","Data":"a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43"} Nov 04 01:03:51.096555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.096438 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5123c16fe97b84c961038527e0e6ea28a1b27635d7231065811f0254562da43" Nov 04 01:03:51.096555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.096519 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71dkjpw" Nov 04 01:03:51.106802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.106586 4486 generic.go:356] "Generic (PLEG): container finished" podID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerID="5552433f7da33d0acb4892ec466b2468a388dcfd8ccceea00129d597b85b6529" exitCode=0 Nov 04 01:03:51.106802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:51.106639 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" event={"ID":"e691e884-950b-43b9-8f7c-7748cc9056ae","Type":"ContainerDied","Data":"5552433f7da33d0acb4892ec466b2468a388dcfd8ccceea00129d597b85b6529"} Nov 04 01:03:52.118563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.118406 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerStarted","Data":"a9fee45cc21546be5c8d694ab58feb731293c7470b07de291c224573a1c5e074"} Nov 04 01:03:52.230878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.230823 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" podStartSLOduration=3.530037419 podStartE2EDuration="35.230808043s" totalImagesPullingTime="31.700770624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:52.228812317 +0000 UTC m=+400.987474478" watchObservedRunningTime="2026-11-04 01:03:52.230808043 +0000 UTC m=+400.989470204" Nov 04 01:03:52.295852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.295823 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:52.338976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.338949 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:52.355650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.355620 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\") pod \"e691e884-950b-43b9-8f7c-7748cc9056ae\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " Nov 04 01:03:52.355797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.355666 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\") pod \"e691e884-950b-43b9-8f7c-7748cc9056ae\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " Nov 04 01:03:52.355797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.355745 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\" (UniqueName: \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\") pod \"e691e884-950b-43b9-8f7c-7748cc9056ae\" (UID: \"e691e884-950b-43b9-8f7c-7748cc9056ae\") " Nov 04 01:03:52.357398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.357362 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle" pod "e691e884-950b-43b9-8f7c-7748cc9056ae" (UID: "e691e884-950b-43b9-8f7c-7748cc9056ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:52.358312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.358284 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2" pod "e691e884-950b-43b9-8f7c-7748cc9056ae" (UID: "e691e884-950b-43b9-8f7c-7748cc9056ae"). InnerVolumeSpecName "kube-api-access-8xdf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:52.362551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.362526 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util" pod "e691e884-950b-43b9-8f7c-7748cc9056ae" (UID: "e691e884-950b-43b9-8f7c-7748cc9056ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:52.456648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.456623 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\" (UniqueName: \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\") pod \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " Nov 04 01:03:52.456787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.456713 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\") pod \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " Nov 04 01:03:52.456787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.456760 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\") pod \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\" (UID: \"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937\") " Nov 04 01:03:52.457411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.457020 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xdf2\" (UniqueName: \"kubernetes.io/projected/e691e884-950b-43b9-8f7c-7748cc9056ae-kube-api-access-8xdf2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:52.457411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.457044 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:52.457411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.457058 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e691e884-950b-43b9-8f7c-7748cc9056ae-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:52.457669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.457631 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle" pod "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" (UID: "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:52.464704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.464420 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx" pod "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" (UID: "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937"). InnerVolumeSpecName "kube-api-access-6rdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:52.464704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.464646 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util" pod "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" (UID: "dafc78b1-6d9b-4717-bd7e-6d7b0ab00937"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:52.557571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.557494 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdwx\" (UniqueName: \"kubernetes.io/projected/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-kube-api-access-6rdwx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:52.557571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.557526 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:52.557571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:52.557541 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dafc78b1-6d9b-4717-bd7e-6d7b0ab00937-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:53.125134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.125051 4486 generic.go:356] "Generic (PLEG): container finished" podID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerID="a9fee45cc21546be5c8d694ab58feb731293c7470b07de291c224573a1c5e074" exitCode=0 Nov 04 01:03:53.125513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.125186 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerDied","Data":"a9fee45cc21546be5c8d694ab58feb731293c7470b07de291c224573a1c5e074"} Nov 04 01:03:53.126893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.126867 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" event={"ID":"e691e884-950b-43b9-8f7c-7748cc9056ae","Type":"ContainerDied","Data":"cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce"} Nov 04 01:03:53.127041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.126892 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnbnkd" Nov 04 01:03:53.127041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.126906 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd53dcf18a617d9d604c7e2b636259775d095849e63a4cdd7c008ba27b4993ce" Nov 04 01:03:53.128794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.128775 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" Nov 04 01:03:53.128794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.128785 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b94ls" event={"ID":"dafc78b1-6d9b-4717-bd7e-6d7b0ab00937","Type":"ContainerDied","Data":"f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c"} Nov 04 01:03:53.128955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.128824 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b4cea74a06c4b429e1d6ece0dcc696a703cc20ee8795c3cb7204384a56173c" Nov 04 01:03:53.130996 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.130963 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerStarted","Data":"94b983fb6725f208c93023e911dc9ecf67927691919ae7d017edb06fe34dc802"} Nov 04 01:03:53.535704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:53.535618 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" podStartSLOduration=4.237117887 podStartE2EDuration="34.535603733s" totalImagesPullingTime="30.298485846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:53.450056211 +0000 UTC m=+402.208718376" watchObservedRunningTime="2026-11-04 01:03:53.535603733 +0000 UTC m=+402.294265893" Nov 04 01:03:54.137071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.136965 4486 generic.go:356] "Generic (PLEG): container finished" podID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerID="94b983fb6725f208c93023e911dc9ecf67927691919ae7d017edb06fe34dc802" exitCode=0 Nov 04 01:03:54.137071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.137022 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerDied","Data":"94b983fb6725f208c93023e911dc9ecf67927691919ae7d017edb06fe34dc802"} Nov 04 01:03:54.139112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.139088 4486 generic.go:356] "Generic (PLEG): container finished" podID="81ef04dd-8c47-47b1-b965-33cab1207136" containerID="3594585cdb8cfb61d3b3ff1da46907ce4e843e18f4e362545db8143f8e439409" exitCode=0 Nov 04 01:03:54.139243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.139163 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" event={"ID":"81ef04dd-8c47-47b1-b965-33cab1207136","Type":"ContainerDied","Data":"3594585cdb8cfb61d3b3ff1da46907ce4e843e18f4e362545db8143f8e439409"} Nov 04 01:03:54.141077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.141054 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerStarted","Data":"9f787f829943661932fa2737f6bd556118b981ead5e4485d543c6d39f5acd2fd"} Nov 04 01:03:54.145964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.145944 4486 generic.go:356] "Generic (PLEG): container finished" podID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerID="8a032e44e06fae3e951a41b84ee536f7d8954d7fee450b64e4aaef2fa1cd4757" exitCode=0 Nov 04 01:03:54.146088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.146011 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" event={"ID":"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8","Type":"ContainerDied","Data":"8a032e44e06fae3e951a41b84ee536f7d8954d7fee450b64e4aaef2fa1cd4757"} Nov 04 01:03:54.148129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.148097 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerStarted","Data":"57bad690c5a6812037ee8d0103a50b3b41d895bd34deb9c1ed82d5e8dc11abe2"} Nov 04 01:03:54.298879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.298857 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:54.359564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.359522 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" podStartSLOduration=3.449811462 podStartE2EDuration="41.359508511s" totalImagesPullingTime="37.909697049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:54.357742883 +0000 UTC m=+403.116405044" watchObservedRunningTime="2026-11-04 01:03:54.359508511 +0000 UTC m=+403.118170671" Nov 04 01:03:54.372486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.372464 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\") pod \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " Nov 04 01:03:54.372542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.372535 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\") pod \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " Nov 04 01:03:54.372587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.372574 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\" (UniqueName: \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\") pod \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\" (UID: \"4523d68a-7f7b-4010-a621-c99e5d7bf9be\") " Nov 04 01:03:54.372913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.372888 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle" pod "4523d68a-7f7b-4010-a621-c99e5d7bf9be" (UID: "4523d68a-7f7b-4010-a621-c99e5d7bf9be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:54.374735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.374715 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh" pod "4523d68a-7f7b-4010-a621-c99e5d7bf9be" (UID: "4523d68a-7f7b-4010-a621-c99e5d7bf9be"). InnerVolumeSpecName "kube-api-access-rxfrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:54.378247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.378226 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util" pod "4523d68a-7f7b-4010-a621-c99e5d7bf9be" (UID: "4523d68a-7f7b-4010-a621-c99e5d7bf9be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:54.468419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.468340 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" podStartSLOduration=4.2796834409999995 podStartE2EDuration="35.468325165s" totalImagesPullingTime="31.188641724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:54.467755625 +0000 UTC m=+403.226417787" watchObservedRunningTime="2026-11-04 01:03:54.468325165 +0000 UTC m=+403.226987325" Nov 04 01:03:54.473576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.473558 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:54.473628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.473579 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfrh\" (UniqueName: \"kubernetes.io/projected/4523d68a-7f7b-4010-a621-c99e5d7bf9be-kube-api-access-rxfrh\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:54.473628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:54.473589 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4523d68a-7f7b-4010-a621-c99e5d7bf9be-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.154351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.154322 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:03:55.155056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.155033 4486 generic.go:356] "Generic (PLEG): container finished" podID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerID="57bad690c5a6812037ee8d0103a50b3b41d895bd34deb9c1ed82d5e8dc11abe2" exitCode=1 Nov 04 01:03:55.155143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.155114 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerDied","Data":"57bad690c5a6812037ee8d0103a50b3b41d895bd34deb9c1ed82d5e8dc11abe2"} Nov 04 01:03:55.157141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.157117 4486 generic.go:356] "Generic (PLEG): container finished" podID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerID="9f787f829943661932fa2737f6bd556118b981ead5e4485d543c6d39f5acd2fd" exitCode=0 Nov 04 01:03:55.157265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.157196 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerDied","Data":"9f787f829943661932fa2737f6bd556118b981ead5e4485d543c6d39f5acd2fd"} Nov 04 01:03:55.159153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.159135 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" Nov 04 01:03:55.159253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.159180 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tj25v" event={"ID":"4523d68a-7f7b-4010-a621-c99e5d7bf9be","Type":"ContainerDied","Data":"5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528"} Nov 04 01:03:55.159253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.159214 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ef7c5b1db35f2e90ffb47ef5a27e8c5c1bcc3abf0e2765a21873159e1dc5528" Nov 04 01:03:55.299400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.299377 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:55.341546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.341525 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:55.344463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.344443 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:55.380441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.380418 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\") pod \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " Nov 04 01:03:55.380532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.380460 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\") pod \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " Nov 04 01:03:55.380532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.380488 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\" (UniqueName: \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\") pod \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\" (UID: \"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1\") " Nov 04 01:03:55.380928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.380901 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle" pod "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" (UID: "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.382452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.382426 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc" pod "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" (UID: "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1"). InnerVolumeSpecName "kube-api-access-xvqrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:55.385660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.385639 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util" pod "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" (UID: "a28b03d5-a3b9-4087-8eda-fe37a9c79bf1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.481808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.481744 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\") pod \"81ef04dd-8c47-47b1-b965-33cab1207136\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " Nov 04 01:03:55.481908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.481804 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\" (UniqueName: \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\") pod \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " Nov 04 01:03:55.481908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.481887 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\") pod \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " Nov 04 01:03:55.482051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.481974 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\") pod \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\" (UID: \"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8\") " Nov 04 01:03:55.482051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482020 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\") pod \"81ef04dd-8c47-47b1-b965-33cab1207136\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " Nov 04 01:03:55.482154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482055 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\" (UniqueName: \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\") pod \"81ef04dd-8c47-47b1-b965-33cab1207136\" (UID: \"81ef04dd-8c47-47b1-b965-33cab1207136\") " Nov 04 01:03:55.482358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482339 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.482438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482362 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.482438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482377 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqrc\" (UniqueName: \"kubernetes.io/projected/a28b03d5-a3b9-4087-8eda-fe37a9c79bf1-kube-api-access-xvqrc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.482555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.482530 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle" pod "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" (UID: "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.483232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.483205 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle" pod "81ef04dd-8c47-47b1-b965-33cab1207136" (UID: "81ef04dd-8c47-47b1-b965-33cab1207136"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.483878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.483853 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj" pod "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" (UID: "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8"). InnerVolumeSpecName "kube-api-access-d44lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:55.484796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.484771 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj" pod "81ef04dd-8c47-47b1-b965-33cab1207136" (UID: "81ef04dd-8c47-47b1-b965-33cab1207136"). InnerVolumeSpecName "kube-api-access-p98qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:55.488504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.488476 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util" pod "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" (UID: "bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.488573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.488521 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util" pod "81ef04dd-8c47-47b1-b965-33cab1207136" (UID: "81ef04dd-8c47-47b1-b965-33cab1207136"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:55.582788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582765 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p98qj\" (UniqueName: \"kubernetes.io/projected/81ef04dd-8c47-47b1-b965-33cab1207136-kube-api-access-p98qj\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.582788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582787 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.582896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582797 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44lj\" (UniqueName: \"kubernetes.io/projected/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-kube-api-access-d44lj\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.582896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582806 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.582896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582814 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:55.582896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:55.582822 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81ef04dd-8c47-47b1-b965-33cab1207136-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.165332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.165295 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" event={"ID":"bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8","Type":"ContainerDied","Data":"a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779"} Nov 04 01:03:56.165762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.165347 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3776f29fc8e0d4fce7c8bd77ae5c1553b8a59b9a75766fbb10ef14c5e23f779" Nov 04 01:03:56.165762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.165303 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644w5rjs" Nov 04 01:03:56.167180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.167153 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" event={"ID":"a28b03d5-a3b9-4087-8eda-fe37a9c79bf1","Type":"ContainerDied","Data":"3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad"} Nov 04 01:03:56.167309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.167209 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b48ac6c7ded04d1c8b78de14eb7b308ffc5ff09a6fd08393848766a1fd594ad" Nov 04 01:03:56.167309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.167169 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vxxcd" Nov 04 01:03:56.168930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.168901 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" event={"ID":"81ef04dd-8c47-47b1-b965-33cab1207136","Type":"ContainerDied","Data":"e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792"} Nov 04 01:03:56.169050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.168937 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anb65t" Nov 04 01:03:56.169050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.168948 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6befd681ffd03c54c676e05ae18a104021095260937f0f8bafd5cfa76e77792" Nov 04 01:03:56.286154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.286129 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:56.323628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.323605 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:03:56.325239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.325222 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:56.386999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.386958 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\" (UniqueName: \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\") pod \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " Nov 04 01:03:56.387085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387043 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\") pod \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " Nov 04 01:03:56.387153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387103 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\") pod \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " Nov 04 01:03:56.387202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387154 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\") pod \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\" (UID: \"29da81f9-6162-4fb5-acd5-06eb5fe72c32\") " Nov 04 01:03:56.387254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387233 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\") pod \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " Nov 04 01:03:56.387306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387277 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\" (UniqueName: \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\") pod \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\" (UID: \"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64\") " Nov 04 01:03:56.387897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387841 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle" pod "29da81f9-6162-4fb5-acd5-06eb5fe72c32" (UID: "29da81f9-6162-4fb5-acd5-06eb5fe72c32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:56.388015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.387890 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle" pod "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" (UID: "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:56.389273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.389255 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v" pod "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" (UID: "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64"). InnerVolumeSpecName "kube-api-access-gsp4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:56.389445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.389425 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq" pod "29da81f9-6162-4fb5-acd5-06eb5fe72c32" (UID: "29da81f9-6162-4fb5-acd5-06eb5fe72c32"). InnerVolumeSpecName "kube-api-access-zl4qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:03:56.393655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.393634 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util" pod "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" (UID: "113c9921-1e9f-42ee-a4d5-ee5f27e3fb64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:56.395028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.395005 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util" pod "29da81f9-6162-4fb5-acd5-06eb5fe72c32" (UID: "29da81f9-6162-4fb5-acd5-06eb5fe72c32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:03:56.487814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487752 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.487814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487777 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsp4v\" (UniqueName: \"kubernetes.io/projected/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-kube-api-access-gsp4v\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.487814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487793 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4qq\" (UniqueName: \"kubernetes.io/projected/29da81f9-6162-4fb5-acd5-06eb5fe72c32-kube-api-access-zl4qq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.487814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487805 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/113c9921-1e9f-42ee-a4d5-ee5f27e3fb64-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.488021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487818 4486 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-util\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:56.488021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:56.487831 4486 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29da81f9-6162-4fb5-acd5-06eb5fe72c32-bundle\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:03:57.174100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.174070 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:03:57.174805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.174780 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" Nov 04 01:03:57.174927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.174797 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85" event={"ID":"29da81f9-6162-4fb5-acd5-06eb5fe72c32","Type":"ContainerDied","Data":"114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3"} Nov 04 01:03:57.174927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.174845 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="114768eb09872af9ae6211186c1162482a5a09f314b6d3643d914bcd1527f0a3" Nov 04 01:03:57.176713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.176694 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" event={"ID":"113c9921-1e9f-42ee-a4d5-ee5f27e3fb64","Type":"ContainerDied","Data":"1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878"} Nov 04 01:03:57.176820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.176723 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c784vw" Nov 04 01:03:57.176820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:03:57.176724 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aee22daced27abf87c2ef93c3c7f67de9fb5f8f40d9d8622e2d67626c2d9878" Nov 04 01:04:03.257386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.257346 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g"] Nov 04 01:04:03.257964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.257941 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="util" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.257968 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="util" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258007 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="extract" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258016 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="extract" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258031 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="util" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258039 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="util" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258059 4486 cpu_manager.go:414] "containerMap: removing container" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="util" Nov 04 01:04:03.258076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258066 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258086 4486 cpu_manager.go:414] "containerMap: removing container" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258093 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258109 4486 cpu_manager.go:414] "containerMap: removing container" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258115 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258129 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258141 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258151 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258158 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258168 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258175 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258193 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258200 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258208 4486 cpu_manager.go:414] "containerMap: removing container" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258215 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258231 4486 cpu_manager.go:414] "containerMap: removing container" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258237 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258245 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258252 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258271 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258277 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258293 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258300 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="pull" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258312 4486 cpu_manager.go:414] "containerMap: removing container" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258319 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258337 4486 cpu_manager.go:414] "containerMap: removing container" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258344 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258358 4486 cpu_manager.go:414] "containerMap: removing container" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258364 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="util" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258388 4486 cpu_manager.go:414] "containerMap: removing container" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="extract" Nov 04 01:04:03.258377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258394 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258413 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258420 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258430 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="util" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258438 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="util" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258453 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258460 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258470 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258476 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258497 4486 cpu_manager.go:414] "containerMap: removing container" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258504 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="pull" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258519 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258526 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258536 4486 cpu_manager.go:414] "containerMap: removing container" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="util" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258542 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="util" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258557 4486 cpu_manager.go:414] "containerMap: removing container" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258563 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258681 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafc78b1-6d9b-4717-bd7e-6d7b0ab00937" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258693 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4523d68a-7f7b-4010-a621-c99e5d7bf9be" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258709 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b84c42-f634-4a83-9a71-62caf03a995d" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258725 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e691e884-950b-43b9-8f7c-7748cc9056ae" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258737 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ef04dd-8c47-47b1-b965-33cab1207136" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258746 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28b03d5-a3b9-4087-8eda-fe37a9c79bf1" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258761 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfbf1c56-0a2f-4e1d-b6f4-75b4980162a8" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258770 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="29da81f9-6162-4fb5-acd5-06eb5fe72c32" containerName="extract" Nov 04 01:04:03.259588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.258786 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="113c9921-1e9f-42ee-a4d5-ee5f27e3fb64" containerName="extract" Nov 04 01:04:03.262094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.262072 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.308035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.308007 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g"] Nov 04 01:04:03.341379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.341352 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdvt\" (UniqueName: \"kubernetes.io/projected/1812c494-e212-48d8-b0d0-cec4bd2ab494-kube-api-access-stdvt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.341491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.341385 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1812c494-e212-48d8-b0d0-cec4bd2ab494-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.353678 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.353655 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:04:03.353777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.353743 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-l5wpq\"" Nov 04 01:04:03.353869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.353856 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 04 01:04:03.442328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.442300 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdvt\" (UniqueName: \"kubernetes.io/projected/1812c494-e212-48d8-b0d0-cec4bd2ab494-kube-api-access-stdvt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.442433 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.442331 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1812c494-e212-48d8-b0d0-cec4bd2ab494-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.442649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.442634 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1812c494-e212-48d8-b0d0-cec4bd2ab494-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.496514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.496492 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdvt\" (UniqueName: \"kubernetes.io/projected/1812c494-e212-48d8-b0d0-cec4bd2ab494-kube-api-access-stdvt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g\" (UID: \"1812c494-e212-48d8-b0d0-cec4bd2ab494\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.572572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.572549 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" Nov 04 01:04:03.991780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:03.991702 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g"] Nov 04 01:04:03.995706 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:03.995682 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1812c494_e212_48d8_b0d0_cec4bd2ab494.slice/crio-6eb3e066d4ccee15eb42a91dada9f3490e091a2b521f3aa03d00264459a8dca1 WatchSource:0}: Error finding container 6eb3e066d4ccee15eb42a91dada9f3490e091a2b521f3aa03d00264459a8dca1: Status 404 returned error can't find the container with id 6eb3e066d4ccee15eb42a91dada9f3490e091a2b521f3aa03d00264459a8dca1 Nov 04 01:04:04.207696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:04.207657 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" event={"ID":"1812c494-e212-48d8-b0d0-cec4bd2ab494","Type":"ContainerStarted","Data":"6eb3e066d4ccee15eb42a91dada9f3490e091a2b521f3aa03d00264459a8dca1"} Nov 04 01:04:07.222421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:07.222336 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" event={"ID":"1812c494-e212-48d8-b0d0-cec4bd2ab494","Type":"ContainerStarted","Data":"6ed73d1818f1b0700b1a6457e4b9fa61d4ea97176cee4e4eb99467e734e1a5f6"} Nov 04 01:04:07.433267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:07.433215 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-g9j7g" podStartSLOduration=1.580569867 podStartE2EDuration="4.433201415s" totalImagesPullingTime="2.852631548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:07.384430131 +0000 UTC m=+416.143092292" watchObservedRunningTime="2026-11-04 01:04:07.433201415 +0000 UTC m=+416.191863578" Nov 04 01:04:10.399431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.399396 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l"] Nov 04 01:04:10.401461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.401438 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:10.451994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.451946 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:04:10.452140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.452088 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:04:10.498684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.498648 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-s7674\"" Nov 04 01:04:10.500656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.500633 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l"] Nov 04 01:04:10.502527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.502505 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7fl\" (UniqueName: \"kubernetes.io/projected/130229ab-0216-44e2-aca7-7affd622750e-kube-api-access-7p7fl\") pod \"authorino-operator-7b7bb4bc77-tvh5l\" (UID: \"130229ab-0216-44e2-aca7-7affd622750e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:10.603126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.603089 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7fl\" (UniqueName: \"kubernetes.io/projected/130229ab-0216-44e2-aca7-7affd622750e-kube-api-access-7p7fl\") pod \"authorino-operator-7b7bb4bc77-tvh5l\" (UID: \"130229ab-0216-44e2-aca7-7affd622750e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:10.659298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.659232 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7fl\" (UniqueName: \"kubernetes.io/projected/130229ab-0216-44e2-aca7-7affd622750e-kube-api-access-7p7fl\") pod \"authorino-operator-7b7bb4bc77-tvh5l\" (UID: \"130229ab-0216-44e2-aca7-7affd622750e\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:10.712003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:10.711958 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:11.077166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:11.077137 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l"] Nov 04 01:04:11.108243 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:11.108216 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130229ab_0216_44e2_aca7_7affd622750e.slice/crio-e0a303641503baa1ffa920e3fd1cb22f30c5832b2c2fe8629c1cba8081793149 WatchSource:0}: Error finding container e0a303641503baa1ffa920e3fd1cb22f30c5832b2c2fe8629c1cba8081793149: Status 404 returned error can't find the container with id e0a303641503baa1ffa920e3fd1cb22f30c5832b2c2fe8629c1cba8081793149 Nov 04 01:04:11.240638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:11.240607 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" event={"ID":"130229ab-0216-44e2-aca7-7affd622750e","Type":"ContainerStarted","Data":"e0a303641503baa1ffa920e3fd1cb22f30c5832b2c2fe8629c1cba8081793149"} Nov 04 01:04:13.831109 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:04:13.831073 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:04:14.952311 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:14.952276 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb"] Nov 04 01:04:14.978030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:14.978003 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:15.030677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.030656 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 01:04:15.078241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.078219 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9849x\"" Nov 04 01:04:15.080006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.079969 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb"] Nov 04 01:04:15.140023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.139977 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzh5\" (UniqueName: \"kubernetes.io/projected/88544e2b-965f-4f08-9dd0-dc80129ccecc-kube-api-access-lrzh5\") pod \"dns-operator-controller-manager-8557cbdb84-ndrjb\" (UID: \"88544e2b-965f-4f08-9dd0-dc80129ccecc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:15.240680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.240615 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzh5\" (UniqueName: \"kubernetes.io/projected/88544e2b-965f-4f08-9dd0-dc80129ccecc-kube-api-access-lrzh5\") pod \"dns-operator-controller-manager-8557cbdb84-ndrjb\" (UID: \"88544e2b-965f-4f08-9dd0-dc80129ccecc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:15.296637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.296614 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzh5\" (UniqueName: \"kubernetes.io/projected/88544e2b-965f-4f08-9dd0-dc80129ccecc-kube-api-access-lrzh5\") pod \"dns-operator-controller-manager-8557cbdb84-ndrjb\" (UID: \"88544e2b-965f-4f08-9dd0-dc80129ccecc\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:15.307036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.307014 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:15.746723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:15.746694 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb"] Nov 04 01:04:15.779358 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:15.779332 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88544e2b_965f_4f08_9dd0_dc80129ccecc.slice/crio-8c338175fbf07c95cde02de58aeb0708023486ad37897a4dc452c8404c2d12b6 WatchSource:0}: Error finding container 8c338175fbf07c95cde02de58aeb0708023486ad37897a4dc452c8404c2d12b6: Status 404 returned error can't find the container with id 8c338175fbf07c95cde02de58aeb0708023486ad37897a4dc452c8404c2d12b6 Nov 04 01:04:16.264679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.264644 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" event={"ID":"88544e2b-965f-4f08-9dd0-dc80129ccecc","Type":"ContainerStarted","Data":"8c338175fbf07c95cde02de58aeb0708023486ad37897a4dc452c8404c2d12b6"} Nov 04 01:04:16.266339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.266305 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" event={"ID":"130229ab-0216-44e2-aca7-7affd622750e","Type":"ContainerStarted","Data":"f17cf4ca7cff11fcbfb43b97c51110ae0273d080a9d21c582d5ce7824daeb646"} Nov 04 01:04:16.266464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.266450 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:16.376766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.376720 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" podStartSLOduration=2.09329044 podStartE2EDuration="6.376707807s" totalImagesPullingTime="4.283417367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:16.375103787 +0000 UTC m=+425.133765948" watchObservedRunningTime="2026-11-04 01:04:16.376707807 +0000 UTC m=+425.135369967" Nov 04 01:04:16.611851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.611820 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz"] Nov 04 01:04:16.651091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.651069 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.704103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704073 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:04:16.704103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704094 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:04:16.704103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704082 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:04:16.704326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704276 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-j4czz\"" Nov 04 01:04:16.704326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704290 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:04:16.704404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.704364 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:04:16.750703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750679 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.750839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750712 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.750839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750747 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.750839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750771 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.751094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750860 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.751094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.750962 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhzw\" (UniqueName: \"kubernetes.io/projected/cf28e05e-d84b-4418-87a5-91529848f965-kube-api-access-wzhzw\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.800291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.800264 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz"] Nov 04 01:04:16.851754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851725 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhzw\" (UniqueName: \"kubernetes.io/projected/cf28e05e-d84b-4418-87a5-91529848f965-kube-api-access-wzhzw\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.851882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851764 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.851882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851787 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.851882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851820 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.851882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851838 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.851882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.851856 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.852312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.852288 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-manifests\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.854513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.854488 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-webhook-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.854601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.854521 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.854684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.854662 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf28e05e-d84b-4418-87a5-91529848f965-tmp\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.854878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.854861 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf28e05e-d84b-4418-87a5-91529848f965-metrics-cert\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:16.906482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:16.906426 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhzw\" (UniqueName: \"kubernetes.io/projected/cf28e05e-d84b-4418-87a5-91529848f965-kube-api-access-wzhzw\") pod \"opendatahub-operator-controller-manager-c655894cb-lvsgz\" (UID: \"cf28e05e-d84b-4418-87a5-91529848f965\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:17.003938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:17.003911 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:17.513931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:17.513900 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz"] Nov 04 01:04:17.514597 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:17.514561 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf28e05e_d84b_4418_87a5_91529848f965.slice/crio-d20702047bb65f38b57adff7ff39ae55163576169f8e30be8adbc5bff5611bca WatchSource:0}: Error finding container d20702047bb65f38b57adff7ff39ae55163576169f8e30be8adbc5bff5611bca: Status 404 returned error can't find the container with id d20702047bb65f38b57adff7ff39ae55163576169f8e30be8adbc5bff5611bca Nov 04 01:04:18.277710 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:18.277671 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" event={"ID":"cf28e05e-d84b-4418-87a5-91529848f965","Type":"ContainerStarted","Data":"d20702047bb65f38b57adff7ff39ae55163576169f8e30be8adbc5bff5611bca"} Nov 04 01:04:20.290514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.290432 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" event={"ID":"88544e2b-965f-4f08-9dd0-dc80129ccecc","Type":"ContainerStarted","Data":"16b07037b7c1a600e385039be20c757ec0c8746ca29753149842829c836ac3c3"} Nov 04 01:04:20.290514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.290498 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:20.481363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.481312 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" podStartSLOduration=2.29503839 podStartE2EDuration="6.481298905s" totalImagesPullingTime="4.186260515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:20.478901646 +0000 UTC m=+429.237563807" watchObservedRunningTime="2026-11-04 01:04:20.481298905 +0000 UTC m=+429.239961065" Nov 04 01:04:20.582173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.582143 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg"] Nov 04 01:04:20.588347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.588317 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.638392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.638241 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-kq7zv\"" Nov 04 01:04:20.638591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.638566 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:04:20.639100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.639073 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 01:04:20.641213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.641183 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg"] Nov 04 01:04:20.682503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.682479 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cb0862-26b5-46f9-9089-62261a42d265-tmp\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.682639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.682549 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnv9p\" (UniqueName: \"kubernetes.io/projected/c4cb0862-26b5-46f9-9089-62261a42d265-kube-api-access-gnv9p\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.784061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.784032 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnv9p\" (UniqueName: \"kubernetes.io/projected/c4cb0862-26b5-46f9-9089-62261a42d265-kube-api-access-gnv9p\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.784236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.784093 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cb0862-26b5-46f9-9089-62261a42d265-tmp\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.784466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.784446 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4cb0862-26b5-46f9-9089-62261a42d265-tmp\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.838913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.838841 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnv9p\" (UniqueName: \"kubernetes.io/projected/c4cb0862-26b5-46f9-9089-62261a42d265-kube-api-access-gnv9p\") pod \"jobset-operator-674cd59dc5-jqgsg\" (UID: \"c4cb0862-26b5-46f9-9089-62261a42d265\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:20.912843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:20.912817 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" Nov 04 01:04:21.283485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:21.283462 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg"] Nov 04 01:04:21.285963 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:21.285934 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4cb0862_26b5_46f9_9089_62261a42d265.slice/crio-eadc602240ebad5f10f64b179880606f43df8bc531c8079f85e39ccf9dcb443d WatchSource:0}: Error finding container eadc602240ebad5f10f64b179880606f43df8bc531c8079f85e39ccf9dcb443d: Status 404 returned error can't find the container with id eadc602240ebad5f10f64b179880606f43df8bc531c8079f85e39ccf9dcb443d Nov 04 01:04:21.297180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:21.297154 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" event={"ID":"c4cb0862-26b5-46f9-9089-62261a42d265","Type":"ContainerStarted","Data":"eadc602240ebad5f10f64b179880606f43df8bc531c8079f85e39ccf9dcb443d"} Nov 04 01:04:22.308904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:22.308868 4486 generic.go:356] "Generic (PLEG): container finished" podID="cf28e05e-d84b-4418-87a5-91529848f965" containerID="245e265e7a409b2f0d3478412a2460982502aa50b3fb005f5e3ff1ab4c247a8d" exitCode=0 Nov 04 01:04:22.309357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:22.308976 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" event={"ID":"cf28e05e-d84b-4418-87a5-91529848f965","Type":"ContainerDied","Data":"245e265e7a409b2f0d3478412a2460982502aa50b3fb005f5e3ff1ab4c247a8d"} Nov 04 01:04:23.224926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.224899 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:23.231133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.231113 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.282017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.281975 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jrbsm\"" Nov 04 01:04:23.293274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.293250 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:23.304669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.304640 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.304773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.304705 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.317824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.317798 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" event={"ID":"cf28e05e-d84b-4418-87a5-91529848f965","Type":"ContainerStarted","Data":"7844f0ad51b2123bf007837b07f4fe95687103544347b4010de5597c92c93991"} Nov 04 01:04:23.318180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.317929 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:23.405777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.405747 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.405918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.405887 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.406390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.406365 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.463899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.463847 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\") pod \"kuadrant-operator-controller-manager-cf8469d7b-bc2qs\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:23.519880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.519827 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" podStartSLOduration=1.99821817 podStartE2EDuration="7.519809317s" totalImagesPullingTime="5.07181308s" totalInitContainerRuntime="449.778067ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:23.517884749 +0000 UTC m=+432.276546949" watchObservedRunningTime="2026-11-04 01:04:23.519809317 +0000 UTC m=+432.278471481" Nov 04 01:04:23.544060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:23.544033 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:24.004994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:24.004952 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:24.007408 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:24.007375 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5575e692_0255_453e_97ab_0a0e78c107b4.slice/crio-54237fdf19c2b8c257625bdf151677ac0f900841856d9b39c7edf4659a65b43d WatchSource:0}: Error finding container 54237fdf19c2b8c257625bdf151677ac0f900841856d9b39c7edf4659a65b43d: Status 404 returned error can't find the container with id 54237fdf19c2b8c257625bdf151677ac0f900841856d9b39c7edf4659a65b43d Nov 04 01:04:24.325838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:24.325789 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" event={"ID":"5575e692-0255-453e-97ab-0a0e78c107b4","Type":"ContainerStarted","Data":"54237fdf19c2b8c257625bdf151677ac0f900841856d9b39c7edf4659a65b43d"} Nov 04 01:04:25.333181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:25.333143 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" event={"ID":"c4cb0862-26b5-46f9-9089-62261a42d265","Type":"ContainerStarted","Data":"279694e4f97e96583b414ef48419c6b0745f64fc918bbabb71bd254f341dfe32"} Nov 04 01:04:25.458192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:25.458147 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jqgsg" podStartSLOduration=2.122773681 podStartE2EDuration="5.458134613s" totalImagesPullingTime="3.335360932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:25.457600787 +0000 UTC m=+434.216262947" watchObservedRunningTime="2026-11-04 01:04:25.458134613 +0000 UTC m=+434.216796773" Nov 04 01:04:27.273333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.273299 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-tvh5l" Nov 04 01:04:27.504201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.504169 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f"] Nov 04 01:04:27.509680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.509652 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:27.539277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.539172 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7-kube-api-access-j5c72\") pod \"limitador-operator-controller-manager-5778bb4449-qt25f\" (UID: \"45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:27.565664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.565639 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f"] Nov 04 01:04:27.568926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.568904 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mfrfk\"" Nov 04 01:04:27.640634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.640599 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7-kube-api-access-j5c72\") pod \"limitador-operator-controller-manager-5778bb4449-qt25f\" (UID: \"45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:27.696594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.696563 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5c72\" (UniqueName: \"kubernetes.io/projected/45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7-kube-api-access-j5c72\") pod \"limitador-operator-controller-manager-5778bb4449-qt25f\" (UID: \"45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:27.823886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:27.823846 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:28.198328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:28.198304 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f"] Nov 04 01:04:28.200360 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:28.200333 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ac63b3_2cc2_4ff4_b7f9_cb03288f56b7.slice/crio-0509527a7e87153fc229b8f8e9f5fd31b3b6df1c58b061f65153a25e51d9f932 WatchSource:0}: Error finding container 0509527a7e87153fc229b8f8e9f5fd31b3b6df1c58b061f65153a25e51d9f932: Status 404 returned error can't find the container with id 0509527a7e87153fc229b8f8e9f5fd31b3b6df1c58b061f65153a25e51d9f932 Nov 04 01:04:28.348540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:28.348502 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" event={"ID":"45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7","Type":"ContainerStarted","Data":"0509527a7e87153fc229b8f8e9f5fd31b3b6df1c58b061f65153a25e51d9f932"} Nov 04 01:04:31.300163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:31.300135 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-ndrjb" Nov 04 01:04:31.365908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:31.365870 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" event={"ID":"5575e692-0255-453e-97ab-0a0e78c107b4","Type":"ContainerStarted","Data":"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5"} Nov 04 01:04:31.366076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:31.365971 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:31.531522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:31.531459 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" podStartSLOduration=2.175021294 podStartE2EDuration="8.531440673s" totalImagesPullingTime="6.356419379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:31.527754839 +0000 UTC m=+440.286417000" watchObservedRunningTime="2026-11-04 01:04:31.531440673 +0000 UTC m=+440.290102837" Nov 04 01:04:32.372668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:32.372632 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" event={"ID":"45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7","Type":"ContainerStarted","Data":"397df54c0c615258938300fbd640d93ff4a4036ef87af44613816ce4ce89e43e"} Nov 04 01:04:32.373120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:32.372716 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:32.483434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:32.483378 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" podStartSLOduration=1.944569822 podStartE2EDuration="5.483360944s" totalImagesPullingTime="3.538791122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:32.483123815 +0000 UTC m=+441.241785976" watchObservedRunningTime="2026-11-04 01:04:32.483360944 +0000 UTC m=+441.242023105" Nov 04 01:04:34.328428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:34.328396 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c655894cb-lvsgz" Nov 04 01:04:42.375353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.375322 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:42.735673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.735599 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:04:42.741105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.741083 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:42.797768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.797743 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:04:42.857143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.857120 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv877\" (UniqueName: \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:42.857256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.857154 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:42.958514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.958489 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv877\" (UniqueName: \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:42.958609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.958522 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:42.958853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:42.958837 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:43.012649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:43.012600 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv877\" (UniqueName: \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\") pod \"kuadrant-operator-controller-manager-678d87766f-7zghx\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:43.054482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:43.054461 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:43.379321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:43.379292 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-qt25f" Nov 04 01:04:43.467639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:43.467618 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:04:43.469627 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:43.469597 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808b6f0_06f4_4ba4_8c82_0b650bb4c99e.slice/crio-b02f91774a8efe33b6507d72da0d661b4c618acd0486445f45df78255a2bbed8 WatchSource:0}: Error finding container b02f91774a8efe33b6507d72da0d661b4c618acd0486445f45df78255a2bbed8: Status 404 returned error can't find the container with id b02f91774a8efe33b6507d72da0d661b4c618acd0486445f45df78255a2bbed8 Nov 04 01:04:43.829671 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:04:43.829633 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:04:44.429813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:44.429780 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" event={"ID":"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e","Type":"ContainerStarted","Data":"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c"} Nov 04 01:04:44.430196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:44.429825 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" event={"ID":"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e","Type":"ContainerStarted","Data":"b02f91774a8efe33b6507d72da0d661b4c618acd0486445f45df78255a2bbed8"} Nov 04 01:04:44.430196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:44.429853 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:44.593953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:44.593899 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" podStartSLOduration=2.593884253 podStartE2EDuration="2.593884253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:44.592246232 +0000 UTC m=+453.350908392" watchObservedRunningTime="2026-11-04 01:04:44.593884253 +0000 UTC m=+453.352546415" Nov 04 01:04:50.459846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.459810 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm"] Nov 04 01:04:50.483609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.483580 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.520817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.520786 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm"] Nov 04 01:04:50.565716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.565690 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lb9jp\"" Nov 04 01:04:50.617813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.617769 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.617813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.617817 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.617844 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxk2p\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-kube-api-access-cxk2p\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.617936 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.618008 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.618048 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.618094 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.618131 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.618231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.618160 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719217 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719258 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719279 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxk2p\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-kube-api-access-cxk2p\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719302 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719328 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719362 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719393 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719432 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719474 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.719758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.719634 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.720077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.720052 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.720245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.720122 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.720245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.720156 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.720691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.720659 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.721950 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.721928 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.722248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.722229 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.776247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.776223 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.776651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.776633 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxk2p\" (UniqueName: \"kubernetes.io/projected/b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b-kube-api-access-cxk2p\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm\" (UID: \"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:50.802886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:50.802861 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:51.237658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.237633 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm"] Nov 04 01:04:51.239297 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:51.239258 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b5d7d1_6e26_4ca5_8f19_400bc9e5121b.slice/crio-6a7dc3bd7969c1d4af2f18a116f670bc47dd7c68f9e24c5a81871a548a4c15da WatchSource:0}: Error finding container 6a7dc3bd7969c1d4af2f18a116f670bc47dd7c68f9e24c5a81871a548a4c15da: Status 404 returned error can't find the container with id 6a7dc3bd7969c1d4af2f18a116f670bc47dd7c68f9e24c5a81871a548a4c15da Nov 04 01:04:51.241605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.241568 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:04:51.241695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.241677 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:04:51.241743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.241724 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:04:51.470043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.470006 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" event={"ID":"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b","Type":"ContainerStarted","Data":"79973944e6709669b2f04ab0b58cc0af3f47f047f2353554d8751d04de5161ff"} Nov 04 01:04:51.470424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.470078 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" event={"ID":"b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b","Type":"ContainerStarted","Data":"6a7dc3bd7969c1d4af2f18a116f670bc47dd7c68f9e24c5a81871a548a4c15da"} Nov 04 01:04:51.611692 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.611644 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" podStartSLOduration=1.61162836 podStartE2EDuration="1.61162836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:51.609222984 +0000 UTC m=+460.367885159" watchObservedRunningTime="2026-11-04 01:04:51.61162836 +0000 UTC m=+460.370290520" Nov 04 01:04:51.803280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:51.803248 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:52.807799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:52.807767 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:53.482940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:53.482911 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:53.483799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:53.483778 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm" Nov 04 01:04:54.078578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.078543 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:04:54.081184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.081160 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:04:54.132150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.132115 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d5p9f\"" Nov 04 01:04:54.134869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.134843 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:04:54.147976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.147954 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mtm\" (UniqueName: \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\") pod \"authorino-57f7f8659f-drjm7\" (UID: \"dbb934f9-aeab-428c-b46c-3cff9f49706c\") " pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:04:54.249027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.248974 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mtm\" (UniqueName: \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\") pod \"authorino-57f7f8659f-drjm7\" (UID: \"dbb934f9-aeab-428c-b46c-3cff9f49706c\") " pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:04:54.303777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.303751 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mtm\" (UniqueName: \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\") pod \"authorino-57f7f8659f-drjm7\" (UID: \"dbb934f9-aeab-428c-b46c-3cff9f49706c\") " pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:04:54.392334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.392265 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:04:54.753253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:54.753227 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:04:54.755498 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:04:54.755466 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbb934f9_aeab_428c_b46c_3cff9f49706c.slice/crio-421230a1de44a902e009e9bac5e384c6dc6c0c8776de34bab72125e0c57335c3 WatchSource:0}: Error finding container 421230a1de44a902e009e9bac5e384c6dc6c0c8776de34bab72125e0c57335c3: Status 404 returned error can't find the container with id 421230a1de44a902e009e9bac5e384c6dc6c0c8776de34bab72125e0c57335c3 Nov 04 01:04:55.436742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:55.436710 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:04:55.493906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:55.493872 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-drjm7" event={"ID":"dbb934f9-aeab-428c-b46c-3cff9f49706c","Type":"ContainerStarted","Data":"421230a1de44a902e009e9bac5e384c6dc6c0c8776de34bab72125e0c57335c3"} Nov 04 01:04:55.648701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:55.648666 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:55.648926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:55.648903 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" podUID="5575e692-0255-453e-97ab-0a0e78c107b4" containerName="manager" containerID="cri-o://d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5" gracePeriod=10 Nov 04 01:04:57.205705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.205682 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:57.275518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.275453 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\") pod \"5575e692-0255-453e-97ab-0a0e78c107b4\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " Nov 04 01:04:57.275654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.275558 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\") pod \"5575e692-0255-453e-97ab-0a0e78c107b4\" (UID: \"5575e692-0255-453e-97ab-0a0e78c107b4\") " Nov 04 01:04:57.276008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.275952 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume" pod "5575e692-0255-453e-97ab-0a0e78c107b4" (UID: "5575e692-0255-453e-97ab-0a0e78c107b4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:04:57.278035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.277976 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql" pod "5575e692-0255-453e-97ab-0a0e78c107b4" (UID: "5575e692-0255-453e-97ab-0a0e78c107b4"). InnerVolumeSpecName "kube-api-access-xhcql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:04:57.376646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.376621 4486 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5575e692-0255-453e-97ab-0a0e78c107b4-extensions-socket-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:57.376782 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.376649 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhcql\" (UniqueName: \"kubernetes.io/projected/5575e692-0255-453e-97ab-0a0e78c107b4-kube-api-access-xhcql\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:04:57.505443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.505410 4486 generic.go:356] "Generic (PLEG): container finished" podID="5575e692-0255-453e-97ab-0a0e78c107b4" containerID="d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5" exitCode=0 Nov 04 01:04:57.505604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.505463 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" Nov 04 01:04:57.505604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.505493 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" event={"ID":"5575e692-0255-453e-97ab-0a0e78c107b4","Type":"ContainerDied","Data":"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5"} Nov 04 01:04:57.505604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.505560 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs" event={"ID":"5575e692-0255-453e-97ab-0a0e78c107b4","Type":"ContainerDied","Data":"54237fdf19c2b8c257625bdf151677ac0f900841856d9b39c7edf4659a65b43d"} Nov 04 01:04:57.505604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.505581 4486 scope.go:122] "RemoveContainer" containerID="d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5" Nov 04 01:04:57.514174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.514156 4486 scope.go:122] "RemoveContainer" containerID="d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5" Nov 04 01:04:57.514413 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:04:57.514397 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5\": container with ID starting with d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5 not found: ID does not exist" containerID="d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5" Nov 04 01:04:57.514481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.514418 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5"} err="failed to get container status \"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5\": rpc error: code = NotFound desc = could not find container \"d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5\": container with ID starting with d80dcc0193ab74f9276304e4a58d51c0164f3a38329a70caca009f9a6e4a13a5 not found: ID does not exist" Nov 04 01:04:57.667054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.666973 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:57.670792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:57.670764 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-bc2qs"] Nov 04 01:04:59.348624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:59.348592 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5575e692-0255-453e-97ab-0a0e78c107b4" path="/var/lib/kubelet/pods/5575e692-0255-453e-97ab-0a0e78c107b4/volumes" Nov 04 01:04:59.517922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:59.517890 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-drjm7" event={"ID":"dbb934f9-aeab-428c-b46c-3cff9f49706c","Type":"ContainerStarted","Data":"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3"} Nov 04 01:04:59.625647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:04:59.625558 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-drjm7" podStartSLOduration=1.737020724 podStartE2EDuration="5.625544305s" totalImagesPullingTime="3.888523581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:04:59.624511033 +0000 UTC m=+468.383173193" watchObservedRunningTime="2026-11-04 01:04:59.625544305 +0000 UTC m=+468.384206494" Nov 04 01:05:13.829572 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:13.829536 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:05:16.964755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:16.964719 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88"] Nov 04 01:05:16.965151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:16.965059 4486 cpu_manager.go:414] "containerMap: removing container" podUID="5575e692-0255-453e-97ab-0a0e78c107b4" containerName="manager" Nov 04 01:05:16.965151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:16.965071 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5575e692-0255-453e-97ab-0a0e78c107b4" containerName="manager" Nov 04 01:05:16.965151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:16.965130 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="5575e692-0255-453e-97ab-0a0e78c107b4" containerName="manager" Nov 04 01:05:16.968124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:16.968105 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.018742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.018707 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:05:17.018881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.018844 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:05:17.066129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.066101 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-4mlgb\"" Nov 04 01:05:17.138383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.138356 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.138537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.138406 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.138537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.138521 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rnx\" (UniqueName: \"kubernetes.io/projected/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-kube-api-access-d6rnx\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.138628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.138559 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.161548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.161508 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 01:05:17.163721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.163697 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88"] Nov 04 01:05:17.239454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239395 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rnx\" (UniqueName: \"kubernetes.io/projected/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-kube-api-access-d6rnx\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.239454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239437 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.239612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239460 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.239612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239492 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.239804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239789 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.239918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.239900 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.240322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.240306 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.298409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.298386 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rnx\" (UniqueName: \"kubernetes.io/projected/33067ffe-3e40-4d78-9fa2-9c8e8c7617b5-kube-api-access-d6rnx\") pod \"kserve-module-controller-manager-7566cf5fc8-mwp88\" (UID: \"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.386530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.386501 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:05:17.389495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.389475 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:05:17.443063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.443041 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:05:17.542019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.541941 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbw7r\" (UniqueName: \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\") pod \"authorino-56c6c9ddd5-5x7ft\" (UID: \"3716571a-02b1-4abd-aec9-977b17f7cd59\") " pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:05:17.580058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.580034 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:17.642576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.642545 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbw7r\" (UniqueName: \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\") pod \"authorino-56c6c9ddd5-5x7ft\" (UID: \"3716571a-02b1-4abd-aec9-977b17f7cd59\") " pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:05:17.697193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.697167 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbw7r\" (UniqueName: \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\") pod \"authorino-56c6c9ddd5-5x7ft\" (UID: \"3716571a-02b1-4abd-aec9-977b17f7cd59\") " pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:05:17.699895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.699881 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:05:17.991495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:17.991467 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88"] Nov 04 01:05:17.992750 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:17.992726 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33067ffe_3e40_4d78_9fa2_9c8e8c7617b5.slice/crio-48f192fdd24a6dcd09b71de81e5a9eee2ba71a45c03d78a0a0fba16065b10758 WatchSource:0}: Error finding container 48f192fdd24a6dcd09b71de81e5a9eee2ba71a45c03d78a0a0fba16065b10758: Status 404 returned error can't find the container with id 48f192fdd24a6dcd09b71de81e5a9eee2ba71a45c03d78a0a0fba16065b10758 Nov 04 01:05:18.095637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:18.095613 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:05:18.096662 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:18.096633 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3716571a_02b1_4abd_aec9_977b17f7cd59.slice/crio-71da2b9881b829c7968c5af9f02bb03ccc117a5384d2ed07f9b3ef24793cc8ec WatchSource:0}: Error finding container 71da2b9881b829c7968c5af9f02bb03ccc117a5384d2ed07f9b3ef24793cc8ec: Status 404 returned error can't find the container with id 71da2b9881b829c7968c5af9f02bb03ccc117a5384d2ed07f9b3ef24793cc8ec Nov 04 01:05:18.611379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:18.611338 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" event={"ID":"3716571a-02b1-4abd-aec9-977b17f7cd59","Type":"ContainerStarted","Data":"71da2b9881b829c7968c5af9f02bb03ccc117a5384d2ed07f9b3ef24793cc8ec"} Nov 04 01:05:18.612565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:18.612547 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" event={"ID":"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5","Type":"ContainerStarted","Data":"48f192fdd24a6dcd09b71de81e5a9eee2ba71a45c03d78a0a0fba16065b10758"} Nov 04 01:05:20.625936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:20.625894 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" event={"ID":"3716571a-02b1-4abd-aec9-977b17f7cd59","Type":"ContainerStarted","Data":"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09"} Nov 04 01:05:20.739245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:20.739181 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" podStartSLOduration=1.899294953 podStartE2EDuration="3.739161293s" totalImagesPullingTime="1.83986634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:20.737114667 +0000 UTC m=+489.495776829" watchObservedRunningTime="2026-11-04 01:05:20.739161293 +0000 UTC m=+489.497823456" Nov 04 01:05:20.775491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:20.775449 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:05:20.775909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:20.775668 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-drjm7" podUID="dbb934f9-aeab-428c-b46c-3cff9f49706c" containerName="authorino" containerID="cri-o://bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3" gracePeriod=30 Nov 04 01:05:21.031300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.031274 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:05:21.148781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.148715 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx"] Nov 04 01:05:21.149945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.149099 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dbb934f9-aeab-428c-b46c-3cff9f49706c" containerName="authorino" Nov 04 01:05:21.149945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.149117 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb934f9-aeab-428c-b46c-3cff9f49706c" containerName="authorino" Nov 04 01:05:21.149945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.149201 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb934f9-aeab-428c-b46c-3cff9f49706c" containerName="authorino" Nov 04 01:05:21.153223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.153192 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.172522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.172493 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\" (UniqueName: \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\") pod \"dbb934f9-aeab-428c-b46c-3cff9f49706c\" (UID: \"dbb934f9-aeab-428c-b46c-3cff9f49706c\") " Nov 04 01:05:21.175311 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.175279 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm" pod "dbb934f9-aeab-428c-b46c-3cff9f49706c" (UID: "dbb934f9-aeab-428c-b46c-3cff9f49706c"). InnerVolumeSpecName "kube-api-access-z8mtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:05:21.206420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.206387 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 04 01:05:21.206537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.206509 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cw489\"" Nov 04 01:05:21.207193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.207173 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:05:21.207193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.207178 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:05:21.208319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.208288 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx"] Nov 04 01:05:21.273592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.273565 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xbf\" (UniqueName: \"kubernetes.io/projected/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-kube-api-access-c7xbf\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.273727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.273705 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-data\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.273819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.273806 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8mtm\" (UniqueName: \"kubernetes.io/projected/dbb934f9-aeab-428c-b46c-3cff9f49706c-kube-api-access-z8mtm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:05:21.375118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.375095 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-data\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.375237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.375145 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xbf\" (UniqueName: \"kubernetes.io/projected/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-kube-api-access-c7xbf\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.375433 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.375413 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-data\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.430906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.430848 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xbf\" (UniqueName: \"kubernetes.io/projected/dc438b0b-8727-463a-ac5e-49b0c22ea2d5-kube-api-access-c7xbf\") pod \"postgres-7cdbf6dc44-b46jx\" (UID: \"dc438b0b-8727-463a-ac5e-49b0c22ea2d5\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.468147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.468115 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:21.632176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.632143 4486 generic.go:356] "Generic (PLEG): container finished" podID="dbb934f9-aeab-428c-b46c-3cff9f49706c" containerID="bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3" exitCode=0 Nov 04 01:05:21.632586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.632185 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-drjm7" Nov 04 01:05:21.632586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.632224 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-drjm7" event={"ID":"dbb934f9-aeab-428c-b46c-3cff9f49706c","Type":"ContainerDied","Data":"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3"} Nov 04 01:05:21.632586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.632266 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-drjm7" event={"ID":"dbb934f9-aeab-428c-b46c-3cff9f49706c","Type":"ContainerDied","Data":"421230a1de44a902e009e9bac5e384c6dc6c0c8776de34bab72125e0c57335c3"} Nov 04 01:05:21.632586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.632286 4486 scope.go:122] "RemoveContainer" containerID="bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3" Nov 04 01:05:21.633947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.633913 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" event={"ID":"33067ffe-3e40-4d78-9fa2-9c8e8c7617b5","Type":"ContainerStarted","Data":"2c04681062a02c7fb732b3e18a89f82cb3b6899f6c8731e69eddfab77c9e2572"} Nov 04 01:05:21.641870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.641851 4486 scope.go:122] "RemoveContainer" containerID="bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3" Nov 04 01:05:21.642278 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:21.642252 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3\": container with ID starting with bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3 not found: ID does not exist" containerID="bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3" Nov 04 01:05:21.642340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.642288 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3"} err="failed to get container status \"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3\": rpc error: code = NotFound desc = could not find container \"bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3\": container with ID starting with bad1fbfe4c957f9d59950b0126b7784526a5fec500f00d38ffffa426584da5d3 not found: ID does not exist" Nov 04 01:05:21.796013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.795910 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" podStartSLOduration=2.526097085 podStartE2EDuration="5.795897144s" totalImagesPullingTime="3.269800059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:21.794482271 +0000 UTC m=+490.553144432" watchObservedRunningTime="2026-11-04 01:05:21.795897144 +0000 UTC m=+490.554559304" Nov 04 01:05:21.833356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.833329 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx"] Nov 04 01:05:21.846107 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:21.846071 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc438b0b_8727_463a_ac5e_49b0c22ea2d5.slice/crio-9d0cff237b70dd1d650250de8197c76cffb144ed4e18f6aaf0f67df7d48b4fa6 WatchSource:0}: Error finding container 9d0cff237b70dd1d650250de8197c76cffb144ed4e18f6aaf0f67df7d48b4fa6: Status 404 returned error can't find the container with id 9d0cff237b70dd1d650250de8197c76cffb144ed4e18f6aaf0f67df7d48b4fa6 Nov 04 01:05:21.899918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.899895 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:05:21.919209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:21.919189 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-drjm7"] Nov 04 01:05:22.639913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:22.639872 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" event={"ID":"dc438b0b-8727-463a-ac5e-49b0c22ea2d5","Type":"ContainerStarted","Data":"9d0cff237b70dd1d650250de8197c76cffb144ed4e18f6aaf0f67df7d48b4fa6"} Nov 04 01:05:22.640971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:22.640954 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:23.347936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:23.347905 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb934f9-aeab-428c-b46c-3cff9f49706c" path="/var/lib/kubelet/pods/dbb934f9-aeab-428c-b46c-3cff9f49706c/volumes" Nov 04 01:05:28.679780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:28.679741 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" event={"ID":"dc438b0b-8727-463a-ac5e-49b0c22ea2d5","Type":"ContainerStarted","Data":"7318453a1313282f7f728d530e626bcfbf16f3ba4f6a72a5de29b5feee483f5b"} Nov 04 01:05:28.680146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:28.679864 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:28.791107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:28.791021 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" podStartSLOduration=1.144493975 podStartE2EDuration="7.791005946s" totalImagesPullingTime="6.646511971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:28.7890889 +0000 UTC m=+497.547751061" watchObservedRunningTime="2026-11-04 01:05:28.791005946 +0000 UTC m=+497.549668109" Nov 04 01:05:34.712403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:34.712369 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b46jx" Nov 04 01:05:41.795534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.795493 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:41.798739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.798716 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.836829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.836806 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.836946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.836863 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.837022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.836949 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.848903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.848879 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 01:05:41.849036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.848961 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 01:05:41.884672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.884653 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4whbk\"" Nov 04 01:05:41.895862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.895844 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 01:05:41.899451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.899429 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:41.937406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.937382 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.937529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.937423 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.937529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.937446 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.939827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.939799 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.939925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.939864 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:41.991568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:41.991545 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\") pod \"maas-controller-579cf7c6d-4pd24\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:42.111470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.111439 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:42.233554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.233522 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:05:42.236295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.236274 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.295837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.295758 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:05:42.341445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.341417 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.341556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.341526 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.341604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.341555 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckk8w\" (UniqueName: \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.442132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.442075 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.442132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.442104 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckk8w\" (UniqueName: \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.442132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.442133 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.444467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.444441 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.444741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.444721 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.474502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.474479 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:42.476141 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:42.476111 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329073a8_0a1d_45f5_a64d_19733f4bf3b9.slice/crio-ac683a0a1daf1193a2518d8e7bf2483d33fd4d96a0f290aa1284e7467cc30aea WatchSource:0}: Error finding container ac683a0a1daf1193a2518d8e7bf2483d33fd4d96a0f290aa1284e7467cc30aea: Status 404 returned error can't find the container with id ac683a0a1daf1193a2518d8e7bf2483d33fd4d96a0f290aa1284e7467cc30aea Nov 04 01:05:42.500142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.500114 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckk8w\" (UniqueName: \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\") pod \"maas-controller-566cd577c-lz4fn\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.548548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.548523 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:42.749812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.749735 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4pd24" event={"ID":"329073a8-0a1d-45f5-a64d-19733f4bf3b9","Type":"ContainerStarted","Data":"ac683a0a1daf1193a2518d8e7bf2483d33fd4d96a0f290aa1284e7467cc30aea"} Nov 04 01:05:42.910097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:42.910073 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:05:42.911828 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:42.911802 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9b71b3_2696_4104_b71d_f2b12dfef6a4.slice/crio-1ce6ff8a3cbc3ccd29fe636b28772f752b6c31ec5cb2c904dd72cdce73a73723 WatchSource:0}: Error finding container 1ce6ff8a3cbc3ccd29fe636b28772f752b6c31ec5cb2c904dd72cdce73a73723: Status 404 returned error can't find the container with id 1ce6ff8a3cbc3ccd29fe636b28772f752b6c31ec5cb2c904dd72cdce73a73723 Nov 04 01:05:43.647455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:43.647427 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-mwp88" Nov 04 01:05:43.756565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:43.756530 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerStarted","Data":"1ce6ff8a3cbc3ccd29fe636b28772f752b6c31ec5cb2c904dd72cdce73a73723"} Nov 04 01:05:43.830168 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:43.830134 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:05:46.776464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:46.776372 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4pd24" event={"ID":"329073a8-0a1d-45f5-a64d-19733f4bf3b9","Type":"ContainerStarted","Data":"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d"} Nov 04 01:05:46.776885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:46.776476 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:46.777879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:46.777853 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerStarted","Data":"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c"} Nov 04 01:05:46.778028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:46.778011 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:46.919794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:46.919741 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-4pd24" podStartSLOduration=2.244994171 podStartE2EDuration="5.919728226s" totalImagesPullingTime="3.674734055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:46.918609969 +0000 UTC m=+515.677272129" watchObservedRunningTime="2026-11-04 01:05:46.919728226 +0000 UTC m=+515.678390387" Nov 04 01:05:47.029745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:47.029615 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-566cd577c-lz4fn" podStartSLOduration=1.542904203 podStartE2EDuration="5.029595065s" totalImagesPullingTime="3.486690862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:05:47.027140386 +0000 UTC m=+515.785802550" watchObservedRunningTime="2026-11-04 01:05:47.029595065 +0000 UTC m=+515.788257226" Nov 04 01:05:49.224837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.224796 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h"] Nov 04 01:05:49.227170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.227151 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.278819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.278777 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 04 01:05:49.278964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.278821 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 04 01:05:49.298689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.298657 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.298796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.298724 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14711a91-e9be-49f9-a6cd-8acc642a1af5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.298844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.298799 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.298878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.298845 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gtz\" (UniqueName: \"kubernetes.io/projected/14711a91-e9be-49f9-a6cd-8acc642a1af5-kube-api-access-98gtz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.326156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.326125 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 04 01:05:49.326267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.326175 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rcdp6\"" Nov 04 01:05:49.328575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.328556 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h"] Nov 04 01:05:49.399914 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.399886 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.400084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.399922 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14711a91-e9be-49f9-a6cd-8acc642a1af5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.400084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.399942 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.400084 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:49.400029 4486 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 04 01:05:49.400084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.400060 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gtz\" (UniqueName: \"kubernetes.io/projected/14711a91-e9be-49f9-a6cd-8acc642a1af5-kube-api-access-98gtz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.400264 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:49.400100 4486 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source podName:14711a91-e9be-49f9-a6cd-8acc642a1af5 nodeName:}" failed. No retries permitted until 2026-11-04 01:05:49.90008432 +0000 UTC m=+518.658746460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" (UID: "14711a91-e9be-49f9-a6cd-8acc642a1af5") : configmap "usage-tenancy-proxy-source" not found Nov 04 01:05:49.400597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.400580 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.402421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.402402 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14711a91-e9be-49f9-a6cd-8acc642a1af5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.454227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.454204 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gtz\" (UniqueName: \"kubernetes.io/projected/14711a91-e9be-49f9-a6cd-8acc642a1af5-kube-api-access-98gtz\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.905269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.905240 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:49.905972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:49.905953 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/14711a91-e9be-49f9-a6cd-8acc642a1af5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-t2z7h\" (UID: \"14711a91-e9be-49f9-a6cd-8acc642a1af5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:50.138467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:50.138429 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:05:50.501233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:50.501204 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h"] Nov 04 01:05:50.502891 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:05:50.502858 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14711a91_e9be_49f9_a6cd_8acc642a1af5.slice/crio-19fbaf8d2412c5ca8fd507c614687fbbd5f8bf110fe2b9b65f765318c9b4d5ae WatchSource:0}: Error finding container 19fbaf8d2412c5ca8fd507c614687fbbd5f8bf110fe2b9b65f765318c9b4d5ae: Status 404 returned error can't find the container with id 19fbaf8d2412c5ca8fd507c614687fbbd5f8bf110fe2b9b65f765318c9b4d5ae Nov 04 01:05:50.799048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:50.798959 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" event={"ID":"14711a91-e9be-49f9-a6cd-8acc642a1af5","Type":"ContainerStarted","Data":"19fbaf8d2412c5ca8fd507c614687fbbd5f8bf110fe2b9b65f765318c9b4d5ae"} Nov 04 01:05:57.786475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:57.786436 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:57.787045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:57.786529 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:05:58.039263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.039006 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:58.040097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.039547 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-4pd24" podUID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" containerName="manager" containerID="cri-o://688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d" gracePeriod=30 Nov 04 01:05:58.469384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.469350 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:58.601510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.601425 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\") pod \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " Nov 04 01:05:58.601510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.601481 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\") pod \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " Nov 04 01:05:58.601730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.601542 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\") pod \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\" (UID: \"329073a8-0a1d-45f5-a64d-19733f4bf3b9\") " Nov 04 01:05:58.604776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.604719 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert" pod "329073a8-0a1d-45f5-a64d-19733f4bf3b9" (UID: "329073a8-0a1d-45f5-a64d-19733f4bf3b9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:05:58.604933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.604846 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs" pod "329073a8-0a1d-45f5-a64d-19733f4bf3b9" (UID: "329073a8-0a1d-45f5-a64d-19733f4bf3b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:05:58.605192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.605169 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv" pod "329073a8-0a1d-45f5-a64d-19733f4bf3b9" (UID: "329073a8-0a1d-45f5-a64d-19733f4bf3b9"). InnerVolumeSpecName "kube-api-access-kqtsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:05:58.703081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.703046 4486 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-metrics-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:05:58.703081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.703078 4486 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/329073a8-0a1d-45f5-a64d-19733f4bf3b9-webhook-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:05:58.703081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.703088 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/329073a8-0a1d-45f5-a64d-19733f4bf3b9-kube-api-access-kqtsv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:05:58.855295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.855207 4486 generic.go:356] "Generic (PLEG): container finished" podID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" containerID="688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d" exitCode=0 Nov 04 01:05:58.855295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.855261 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4pd24" Nov 04 01:05:58.855787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.855298 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4pd24" event={"ID":"329073a8-0a1d-45f5-a64d-19733f4bf3b9","Type":"ContainerDied","Data":"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d"} Nov 04 01:05:58.855787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.855372 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4pd24" event={"ID":"329073a8-0a1d-45f5-a64d-19733f4bf3b9","Type":"ContainerDied","Data":"ac683a0a1daf1193a2518d8e7bf2483d33fd4d96a0f290aa1284e7467cc30aea"} Nov 04 01:05:58.855787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.855404 4486 scope.go:122] "RemoveContainer" containerID="688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d" Nov 04 01:05:58.875024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.874970 4486 scope.go:122] "RemoveContainer" containerID="688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d" Nov 04 01:05:58.875371 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:05:58.875334 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d\": container with ID starting with 688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d not found: ID does not exist" containerID="688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d" Nov 04 01:05:58.875484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:58.875380 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d"} err="failed to get container status \"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d\": rpc error: code = NotFound desc = could not find container \"688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d\": container with ID starting with 688d912a4e31b3ef4485198e0fa661745d7295a6073cf89f36d89762aa5f7c2d not found: ID does not exist" Nov 04 01:05:59.012344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:59.012305 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:59.060406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:59.060366 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4pd24"] Nov 04 01:05:59.354567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:05:59.354529 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" path="/var/lib/kubelet/pods/329073a8-0a1d-45f5-a64d-19733f4bf3b9/volumes" Nov 04 01:06:07.912850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:07.912812 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" event={"ID":"14711a91-e9be-49f9-a6cd-8acc642a1af5","Type":"ContainerStarted","Data":"63dc380c4a1c7960ecc5475dfab9c83412b3b20584f6b0493d39f0897d9c45b0"} Nov 04 01:06:07.913311 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:07.912911 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:06:08.024747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:08.024643 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" podStartSLOduration=1.75444368 podStartE2EDuration="19.024628721s" totalImagesPullingTime="17.270185041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:08.023810301 +0000 UTC m=+536.782472454" watchObservedRunningTime="2026-11-04 01:06:08.024628721 +0000 UTC m=+536.783290882" Nov 04 01:06:13.830541 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:06:13.830500 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:06:18.924188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:18.924152 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-t2z7h" Nov 04 01:06:26.006673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.006634 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6x8gq"] Nov 04 01:06:26.007156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.007055 4486 cpu_manager.go:414] "containerMap: removing container" podUID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" containerName="manager" Nov 04 01:06:26.007156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.007072 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" containerName="manager" Nov 04 01:06:26.007156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.007151 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="329073a8-0a1d-45f5-a64d-19733f4bf3b9" containerName="manager" Nov 04 01:06:26.009862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.009840 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.062068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.061967 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6x8gq"] Nov 04 01:06:26.074329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.074290 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dnxx\" (UniqueName: \"kubernetes.io/projected/35775c31-4459-4fe9-beda-839483a09ef4-kube-api-access-9dnxx\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.074511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.074338 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35775c31-4459-4fe9-beda-839483a09ef4-cert\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.106266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.106231 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-d8cbj\"" Nov 04 01:06:26.106266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.106231 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 04 01:06:26.175375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.175331 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dnxx\" (UniqueName: \"kubernetes.io/projected/35775c31-4459-4fe9-beda-839483a09ef4-kube-api-access-9dnxx\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.175375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.175377 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35775c31-4459-4fe9-beda-839483a09ef4-cert\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.178244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.178210 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35775c31-4459-4fe9-beda-839483a09ef4-cert\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.257185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.257091 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dnxx\" (UniqueName: \"kubernetes.io/projected/35775c31-4459-4fe9-beda-839483a09ef4-kube-api-access-9dnxx\") pod \"kserve-controller-manager-776cc66d47-6x8gq\" (UID: \"35775c31-4459-4fe9-beda-839483a09ef4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.322905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.322861 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:26.717387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:26.717361 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6x8gq"] Nov 04 01:06:26.719344 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:06:26.719314 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35775c31_4459_4fe9_beda_839483a09ef4.slice/crio-f8246e499fa180852591e6b22c88955cb169b437a70d18579f94d679c44fb083 WatchSource:0}: Error finding container f8246e499fa180852591e6b22c88955cb169b437a70d18579f94d679c44fb083: Status 404 returned error can't find the container with id f8246e499fa180852591e6b22c88955cb169b437a70d18579f94d679c44fb083 Nov 04 01:06:27.017411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:27.017313 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" event={"ID":"35775c31-4459-4fe9-beda-839483a09ef4","Type":"ContainerStarted","Data":"f8246e499fa180852591e6b22c88955cb169b437a70d18579f94d679c44fb083"} Nov 04 01:06:31.065835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:31.065797 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" event={"ID":"35775c31-4459-4fe9-beda-839483a09ef4","Type":"ContainerStarted","Data":"d2e7725451943398118d74bf07774a78bdecdbcd972402914a17779f3f5990b8"} Nov 04 01:06:31.066355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:31.065971 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:06:31.178151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:31.178090 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" podStartSLOduration=2.893421525 podStartE2EDuration="6.178074531s" totalImagesPullingTime="3.284653006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:31.174936855 +0000 UTC m=+559.933599016" watchObservedRunningTime="2026-11-04 01:06:31.178074531 +0000 UTC m=+559.936736691" Nov 04 01:06:33.527295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.527260 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2x74h"] Nov 04 01:06:33.665909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.665868 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2x74h"] Nov 04 01:06:33.666101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.666066 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:33.748385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.748350 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv26b\" (UniqueName: \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\") pod \"authorino-6f6944bcdd-2x74h\" (UID: \"5a44949b-5734-4484-8fd8-e65a628b7de7\") " pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:33.849838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.849792 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv26b\" (UniqueName: \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\") pod \"authorino-6f6944bcdd-2x74h\" (UID: \"5a44949b-5734-4484-8fd8-e65a628b7de7\") " pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:33.881347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.881292 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:06:33.881529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.881383 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:06:33.885736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.885691 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2x74h"] Nov 04 01:06:33.885961 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:06:33.885940 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jv26b], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-2x74h" podUID="5a44949b-5734-4484-8fd8-e65a628b7de7" Nov 04 01:06:33.907747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.907708 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv26b\" (UniqueName: \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\") pod \"authorino-6f6944bcdd-2x74h\" (UID: \"5a44949b-5734-4484-8fd8-e65a628b7de7\") " pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:33.913517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.912191 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:33.970559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.970517 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:33.970748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:33.970644 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:34.052664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.052627 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdrd\" (UniqueName: \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\") pod \"authorino-6d9588b9d6-8fzqv\" (UID: \"7f3bf63e-8e26-4723-b258-e6210b3e9ea0\") " pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:34.082585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.082547 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:34.092053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.092025 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:34.154001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.153907 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\" (UniqueName: \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\") pod \"5a44949b-5734-4484-8fd8-e65a628b7de7\" (UID: \"5a44949b-5734-4484-8fd8-e65a628b7de7\") " Nov 04 01:06:34.154167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.154100 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdrd\" (UniqueName: \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\") pod \"authorino-6d9588b9d6-8fzqv\" (UID: \"7f3bf63e-8e26-4723-b258-e6210b3e9ea0\") " pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:34.156102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.156070 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b" pod "5a44949b-5734-4484-8fd8-e65a628b7de7" (UID: "5a44949b-5734-4484-8fd8-e65a628b7de7"). InnerVolumeSpecName "kube-api-access-jv26b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:34.208657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.208622 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdrd\" (UniqueName: \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\") pod \"authorino-6d9588b9d6-8fzqv\" (UID: \"7f3bf63e-8e26-4723-b258-e6210b3e9ea0\") " pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:34.255400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.255359 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv26b\" (UniqueName: \"kubernetes.io/projected/5a44949b-5734-4484-8fd8-e65a628b7de7-kube-api-access-jv26b\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:34.321564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.321519 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:34.357916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.357874 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:34.384843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.384802 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:06:34.472389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.472345 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:06:34.472605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.472466 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.522403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.522364 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 01:06:34.556741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.556707 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.621050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.556868 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzd99\" (UniqueName: \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.657661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.657620 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzd99\" (UniqueName: \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.657842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.657758 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.660245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.660217 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.711735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.711666 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzd99\" (UniqueName: \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\") pod \"authorino-69b47595b-8djwq\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.797978 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.797877 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:06:34.809846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:34.809819 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:34.812117 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:06:34.812085 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f3bf63e_8e26_4723_b258_e6210b3e9ea0.slice/crio-c56f42bdb8102371371f0e99feaff1ffb05ead5ab13f8d179ecc2a0a49f16eba WatchSource:0}: Error finding container c56f42bdb8102371371f0e99feaff1ffb05ead5ab13f8d179ecc2a0a49f16eba: Status 404 returned error can't find the container with id c56f42bdb8102371371f0e99feaff1ffb05ead5ab13f8d179ecc2a0a49f16eba Nov 04 01:06:35.089307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.089270 4486 generic.go:356] "Generic (PLEG): container finished" podID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerID="d7406a3eb9a46d23c655a0b9e1bab43429ed5d377d16b933795ff3b86104ee1b" exitCode=0 Nov 04 01:06:35.089520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.089344 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerDied","Data":"d7406a3eb9a46d23c655a0b9e1bab43429ed5d377d16b933795ff3b86104ee1b"} Nov 04 01:06:35.089842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.089819 4486 scope.go:122] "RemoveContainer" containerID="d7406a3eb9a46d23c655a0b9e1bab43429ed5d377d16b933795ff3b86104ee1b" Nov 04 01:06:35.090954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.090930 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" event={"ID":"7f3bf63e-8e26-4723-b258-e6210b3e9ea0","Type":"ContainerStarted","Data":"c56f42bdb8102371371f0e99feaff1ffb05ead5ab13f8d179ecc2a0a49f16eba"} Nov 04 01:06:35.090954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.090947 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-2x74h" Nov 04 01:06:35.091124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.091093 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:06:35.091202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.091182 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:06:35.091248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.091233 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:06:35.186786 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:06:35.186740 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6390c4c8_be4c_4a1e_854b_8471112fc3a5.slice/crio-cf3ba9db5153036a5cde1f4f50e49f5caf778a2320e91f19105431e3bd4cd674 WatchSource:0}: Error finding container cf3ba9db5153036a5cde1f4f50e49f5caf778a2320e91f19105431e3bd4cd674: Status 404 returned error can't find the container with id cf3ba9db5153036a5cde1f4f50e49f5caf778a2320e91f19105431e3bd4cd674 Nov 04 01:06:35.189158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.189136 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:06:35.365801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.365705 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2x74h"] Nov 04 01:06:35.367990 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:35.367950 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-2x74h"] Nov 04 01:06:36.098188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:36.098152 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerStarted","Data":"caf8826158a89a2c5b5f94fba5b56665faa6ccdffe04723c2c28c2e4978d6aa1"} Nov 04 01:06:36.099332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:36.099310 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b47595b-8djwq" event={"ID":"6390c4c8-be4c-4a1e-854b-8471112fc3a5","Type":"ContainerStarted","Data":"cf3ba9db5153036a5cde1f4f50e49f5caf778a2320e91f19105431e3bd4cd674"} Nov 04 01:06:36.610326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:36.610285 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:06:36.610525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:36.610506 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 04 01:06:36.610588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:36.610555 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 04 01:06:37.108466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.108420 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" event={"ID":"7f3bf63e-8e26-4723-b258-e6210b3e9ea0","Type":"ContainerStarted","Data":"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5"} Nov 04 01:06:37.108865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.108711 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" podUID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" containerName="authorino" containerID="cri-o://2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5" gracePeriod=30 Nov 04 01:06:37.217655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.217600 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" podStartSLOduration=2.059816339 podStartE2EDuration="4.217585449s" totalImagesPullingTime="2.15776911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:37.216675654 +0000 UTC m=+565.975337814" watchObservedRunningTime="2026-11-04 01:06:37.217585449 +0000 UTC m=+565.976247609" Nov 04 01:06:37.360344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.360310 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a44949b-5734-4484-8fd8-e65a628b7de7" path="/var/lib/kubelet/pods/5a44949b-5734-4484-8fd8-e65a628b7de7/volumes" Nov 04 01:06:37.576778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.576745 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:37.611079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.611035 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 04 01:06:37.611257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.611121 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 04 01:06:37.687349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.687255 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\" (UniqueName: \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\") pod \"7f3bf63e-8e26-4723-b258-e6210b3e9ea0\" (UID: \"7f3bf63e-8e26-4723-b258-e6210b3e9ea0\") " Nov 04 01:06:37.690062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.690027 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd" pod "7f3bf63e-8e26-4723-b258-e6210b3e9ea0" (UID: "7f3bf63e-8e26-4723-b258-e6210b3e9ea0"). InnerVolumeSpecName "kube-api-access-vhdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:37.789458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:37.789350 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdrd\" (UniqueName: \"kubernetes.io/projected/7f3bf63e-8e26-4723-b258-e6210b3e9ea0-kube-api-access-vhdrd\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:38.115911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.115870 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b47595b-8djwq" event={"ID":"6390c4c8-be4c-4a1e-854b-8471112fc3a5","Type":"ContainerStarted","Data":"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd"} Nov 04 01:06:38.117364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.117335 4486 generic.go:356] "Generic (PLEG): container finished" podID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" containerID="2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5" exitCode=0 Nov 04 01:06:38.117503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.117371 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" Nov 04 01:06:38.117503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.117373 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" event={"ID":"7f3bf63e-8e26-4723-b258-e6210b3e9ea0","Type":"ContainerDied","Data":"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5"} Nov 04 01:06:38.117634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.117510 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9588b9d6-8fzqv" event={"ID":"7f3bf63e-8e26-4723-b258-e6210b3e9ea0","Type":"ContainerDied","Data":"c56f42bdb8102371371f0e99feaff1ffb05ead5ab13f8d179ecc2a0a49f16eba"} Nov 04 01:06:38.117634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.117545 4486 scope.go:122] "RemoveContainer" containerID="2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5" Nov 04 01:06:38.127110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.127082 4486 scope.go:122] "RemoveContainer" containerID="2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5" Nov 04 01:06:38.127456 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:06:38.127431 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5\": container with ID starting with 2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5 not found: ID does not exist" containerID="2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5" Nov 04 01:06:38.127536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.127466 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5"} err="failed to get container status \"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5\": rpc error: code = NotFound desc = could not find container \"2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5\": container with ID starting with 2bbd2512f87fd05383f50d43cbb3c6408602b9ee5b0438dadd5b3bf5d9a0e4c5 not found: ID does not exist" Nov 04 01:06:38.260028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.254946 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69b47595b-8djwq" podStartSLOduration=2.162230215 podStartE2EDuration="4.254921348s" totalImagesPullingTime="2.092691133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:06:38.251218309 +0000 UTC m=+567.009880470" watchObservedRunningTime="2026-11-04 01:06:38.254921348 +0000 UTC m=+567.013583511" Nov 04 01:06:38.285152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.282955 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:06:38.285152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.283188 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" podUID="3716571a-02b1-4abd-aec9-977b17f7cd59" containerName="authorino" containerID="cri-o://b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09" gracePeriod=30 Nov 04 01:06:38.361611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.361572 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:38.363866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.363842 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d9588b9d6-8fzqv"] Nov 04 01:06:38.610952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.610923 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 04 01:06:38.611182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.610971 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 04 01:06:38.642812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.638199 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:06:38.698952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.698920 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\" (UniqueName: \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\") pod \"3716571a-02b1-4abd-aec9-977b17f7cd59\" (UID: \"3716571a-02b1-4abd-aec9-977b17f7cd59\") " Nov 04 01:06:38.701141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.701113 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r" pod "3716571a-02b1-4abd-aec9-977b17f7cd59" (UID: "3716571a-02b1-4abd-aec9-977b17f7cd59"). InnerVolumeSpecName "kube-api-access-mbw7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:06:38.799675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:38.799636 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbw7r\" (UniqueName: \"kubernetes.io/projected/3716571a-02b1-4abd-aec9-977b17f7cd59-kube-api-access-mbw7r\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:06:39.124390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.124353 4486 generic.go:356] "Generic (PLEG): container finished" podID="3716571a-02b1-4abd-aec9-977b17f7cd59" containerID="b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09" exitCode=0 Nov 04 01:06:39.124843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.124394 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" Nov 04 01:06:39.124843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.124441 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" event={"ID":"3716571a-02b1-4abd-aec9-977b17f7cd59","Type":"ContainerDied","Data":"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09"} Nov 04 01:06:39.124843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.124483 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5x7ft" event={"ID":"3716571a-02b1-4abd-aec9-977b17f7cd59","Type":"ContainerDied","Data":"71da2b9881b829c7968c5af9f02bb03ccc117a5384d2ed07f9b3ef24793cc8ec"} Nov 04 01:06:39.124843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.124505 4486 scope.go:122] "RemoveContainer" containerID="b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09" Nov 04 01:06:39.158963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.158940 4486 scope.go:122] "RemoveContainer" containerID="b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09" Nov 04 01:06:39.159306 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:06:39.159285 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09\": container with ID starting with b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09 not found: ID does not exist" containerID="b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09" Nov 04 01:06:39.159376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.159317 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09"} err="failed to get container status \"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09\": rpc error: code = NotFound desc = could not find container \"b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09\": container with ID starting with b4eade3244279abcb9fa774c1599b87ffea5a85ddc8b131d675b6524c4f0ab09 not found: ID does not exist" Nov 04 01:06:39.281457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.281416 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:06:39.284665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.284634 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5x7ft"] Nov 04 01:06:39.355326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.355293 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3716571a-02b1-4abd-aec9-977b17f7cd59" path="/var/lib/kubelet/pods/3716571a-02b1-4abd-aec9-977b17f7cd59/volumes" Nov 04 01:06:39.355640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.355626 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" path="/var/lib/kubelet/pods/7f3bf63e-8e26-4723-b258-e6210b3e9ea0/volumes" Nov 04 01:06:39.610806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.610773 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 04 01:06:39.610968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:39.610827 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 04 01:06:40.610626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:40.610592 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Nov 04 01:06:40.611055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:40.610648 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Nov 04 01:06:41.613244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:41.613217 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:06:41.613612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:41.613475 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:06:41.614073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:06:41.614049 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:06:43.828607 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:06:43.828572 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:07:02.075408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:07:02.075334 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6x8gq" Nov 04 01:07:11.374177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:07:11.374151 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:07:11.377897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:07:11.377870 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:07:11.394133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:07:11.394110 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:07:11.398413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:07:11.398393 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:07:13.829228 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:07:13.829193 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:07:43.830507 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:07:43.830467 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:08:13.829260 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:08:13.829224 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:08:16.654761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:16.654734 4486 generic.go:356] "Generic (PLEG): container finished" podID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerID="e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c" exitCode=1 Nov 04 01:08:16.655118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:16.654805 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerDied","Data":"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c"} Nov 04 01:08:16.655161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:16.655144 4486 scope.go:122] "RemoveContainer" containerID="e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c" Nov 04 01:08:16.655512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:16.655497 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:08:17.784627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:17.784600 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:08:18.666509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:18.666477 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerStarted","Data":"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b"} Nov 04 01:08:18.666709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:18.666567 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:08:29.673603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:29.673519 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:08:34.104868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.104835 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8"] Nov 04 01:08:34.105281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105231 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3716571a-02b1-4abd-aec9-977b17f7cd59" containerName="authorino" Nov 04 01:08:34.105281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105244 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3716571a-02b1-4abd-aec9-977b17f7cd59" containerName="authorino" Nov 04 01:08:34.105281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105263 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" containerName="authorino" Nov 04 01:08:34.105281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105268 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" containerName="authorino" Nov 04 01:08:34.105430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105328 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3716571a-02b1-4abd-aec9-977b17f7cd59" containerName="authorino" Nov 04 01:08:34.105430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.105338 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3bf63e-8e26-4723-b258-e6210b3e9ea0" containerName="authorino" Nov 04 01:08:34.108387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.108369 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.160242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.160189 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:08:34.207469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.207441 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:08:34.207578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.207479 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdglk\"" Nov 04 01:08:34.207628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.207600 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 01:08:34.209722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.209701 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8"] Nov 04 01:08:34.220854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.220826 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.220953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.220856 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.220953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.220884 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.220953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.220945 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6kl\" (UniqueName: \"kubernetes.io/projected/c0d70473-bec3-4c76-95c2-7227ed44617b-kube-api-access-jg6kl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.221153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.221062 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.221153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.221105 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d70473-bec3-4c76-95c2-7227ed44617b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322342 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d70473-bec3-4c76-95c2-7227ed44617b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322414 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322437 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322466 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322584 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6kl\" (UniqueName: \"kubernetes.io/projected/c0d70473-bec3-4c76-95c2-7227ed44617b-kube-api-access-jg6kl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322678 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.322823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322801 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.323024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322873 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.323111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.322939 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.324659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.324639 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0d70473-bec3-4c76-95c2-7227ed44617b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.324733 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.324716 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0d70473-bec3-4c76-95c2-7227ed44617b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.378017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.377937 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6kl\" (UniqueName: \"kubernetes.io/projected/c0d70473-bec3-4c76-95c2-7227ed44617b-kube-api-access-jg6kl\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8\" (UID: \"c0d70473-bec3-4c76-95c2-7227ed44617b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.420839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.420816 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:34.805699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:34.805582 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8"] Nov 04 01:08:34.807952 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:08:34.807926 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d70473_bec3_4c76_95c2_7227ed44617b.slice/crio-5fc07b938f96e162c6eb8f1c44812b93d4fa410a79dccfe4a07a3ea69c5c23b1 WatchSource:0}: Error finding container 5fc07b938f96e162c6eb8f1c44812b93d4fa410a79dccfe4a07a3ea69c5c23b1: Status 404 returned error can't find the container with id 5fc07b938f96e162c6eb8f1c44812b93d4fa410a79dccfe4a07a3ea69c5c23b1 Nov 04 01:08:35.753746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:35.753706 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" event={"ID":"c0d70473-bec3-4c76-95c2-7227ed44617b","Type":"ContainerStarted","Data":"5fc07b938f96e162c6eb8f1c44812b93d4fa410a79dccfe4a07a3ea69c5c23b1"} Nov 04 01:08:37.768215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:37.768177 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" event={"ID":"c0d70473-bec3-4c76-95c2-7227ed44617b","Type":"ContainerStarted","Data":"f84b90d2b447a235c94a3811f6496af28a615590ea05b22d0d2de37ef0bbc5b7"} Nov 04 01:08:37.877548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:37.877498 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" podStartSLOduration=1.7709834880000002 podStartE2EDuration="3.877485094s" totalImagesPullingTime="2.106501606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:37.876710276 +0000 UTC m=+686.635372441" watchObservedRunningTime="2026-11-04 01:08:37.877485094 +0000 UTC m=+686.636147298" Nov 04 01:08:37.982504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:37.982472 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zrnhb"] Nov 04 01:08:37.985946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:37.985921 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.038936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.038874 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 01:08:38.055303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.055273 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zrnhb"] Nov 04 01:08:38.154816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.154782 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hj9\" (UniqueName: \"kubernetes.io/projected/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-kube-api-access-m9hj9\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.154979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.154827 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.154979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.154917 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.154979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.154949 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.154979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.154974 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.155182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.155098 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256183 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hj9\" (UniqueName: \"kubernetes.io/projected/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-kube-api-access-m9hj9\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256244 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256291 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256320 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256351 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256388 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256790 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-home\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.256917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.256803 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.257130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.257108 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.258518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.258489 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.258700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.258681 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.314587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.314564 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hj9\" (UniqueName: \"kubernetes.io/projected/344cadfc-a8d8-4ced-88d9-c046e7ecd3e9-kube-api-access-m9hj9\") pod \"echo-simulator-kserve-64b6f8f96c-zrnhb\" (UID: \"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.421535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.421502 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:38.600409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.600337 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:38.965257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:38.965234 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zrnhb"] Nov 04 01:08:38.966817 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:08:38.966784 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344cadfc_a8d8_4ced_88d9_c046e7ecd3e9.slice/crio-6bb40cbf3586628ec087e8a48622bf2f64d5ba0922887f66c7acbe414f2f2545 WatchSource:0}: Error finding container 6bb40cbf3586628ec087e8a48622bf2f64d5ba0922887f66c7acbe414f2f2545: Status 404 returned error can't find the container with id 6bb40cbf3586628ec087e8a48622bf2f64d5ba0922887f66c7acbe414f2f2545 Nov 04 01:08:39.785215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:39.785180 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" event={"ID":"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9","Type":"ContainerStarted","Data":"6bb40cbf3586628ec087e8a48622bf2f64d5ba0922887f66c7acbe414f2f2545"} Nov 04 01:08:41.798416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:41.798373 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" event={"ID":"344cadfc-a8d8-4ced-88d9-c046e7ecd3e9","Type":"ContainerStarted","Data":"e7ce56d2938a6b13584c06946fc4ab1150e71991f2c6951ab883071c2ee9e904"} Nov 04 01:08:41.920146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:41.920033 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" podStartSLOduration=2.231630437 podStartE2EDuration="4.920019295s" totalImagesPullingTime="2.688388858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:41.915687125 +0000 UTC m=+690.674349285" watchObservedRunningTime="2026-11-04 01:08:41.920019295 +0000 UTC m=+690.678681456" Nov 04 01:08:42.600793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:42.600755 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:43.830367 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:08:43.830326 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:08:44.421487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:44.421441 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:44.434339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:44.434306 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:44.830558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:44.830525 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" Nov 04 01:08:48.202319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.202285 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7"] Nov 04 01:08:48.205539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.205517 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.310610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.310569 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 01:08:48.313156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.313127 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7"] Nov 04 01:08:48.364606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.364554 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.364606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.364608 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.364866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.364736 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.364958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.364935 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.365041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.364976 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwcr\" (UniqueName: \"kubernetes.io/projected/fec97081-9922-47be-8eb7-3d25b8500e75-kube-api-access-jkwcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.365104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.365053 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec97081-9922-47be-8eb7-3d25b8500e75-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.465925 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.465966 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466020 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466075 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466093 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwcr\" (UniqueName: \"kubernetes.io/projected/fec97081-9922-47be-8eb7-3d25b8500e75-kube-api-access-jkwcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466134 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec97081-9922-47be-8eb7-3d25b8500e75-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466378 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466497 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.466653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.466515 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.468303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.468276 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec97081-9922-47be-8eb7-3d25b8500e75-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.468579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.468561 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec97081-9922-47be-8eb7-3d25b8500e75-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.520796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.520756 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwcr\" (UniqueName: \"kubernetes.io/projected/fec97081-9922-47be-8eb7-3d25b8500e75-kube-api-access-jkwcr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7\" (UID: \"fec97081-9922-47be-8eb7-3d25b8500e75\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.600689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.600634 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:48.618101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.618064 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:48.817783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.817748 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:48.849517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:48.849486 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" Nov 04 01:08:49.188035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:49.188009 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7"] Nov 04 01:08:49.189893 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:08:49.189862 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec97081_9922_47be_8eb7_3d25b8500e75.slice/crio-ecf6249a89c9b1e92f5d83d5bc48ca1af4c4ec891373b8f70a858b044cebdb6d WatchSource:0}: Error finding container ecf6249a89c9b1e92f5d83d5bc48ca1af4c4ec891373b8f70a858b044cebdb6d: Status 404 returned error can't find the container with id ecf6249a89c9b1e92f5d83d5bc48ca1af4c4ec891373b8f70a858b044cebdb6d Nov 04 01:08:49.845303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:49.845263 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" event={"ID":"fec97081-9922-47be-8eb7-3d25b8500e75","Type":"ContainerStarted","Data":"ecf6249a89c9b1e92f5d83d5bc48ca1af4c4ec891373b8f70a858b044cebdb6d"} Nov 04 01:08:50.853057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:50.853023 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" event={"ID":"fec97081-9922-47be-8eb7-3d25b8500e75","Type":"ContainerStarted","Data":"f7662bbe0d4618950d61297f88aebcf3586f4ec0aa7cf98fff2fb2202685223c"} Nov 04 01:08:50.963947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:50.963890 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" podStartSLOduration=2.34533563 podStartE2EDuration="2.963876459s" totalImagesPullingTime="618.540829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:50.96178538 +0000 UTC m=+699.720447540" watchObservedRunningTime="2026-11-04 01:08:50.963876459 +0000 UTC m=+699.722538619" Nov 04 01:08:51.818549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:51.818510 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:56.336563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.336523 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq"] Nov 04 01:08:56.341552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.341500 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.398386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.398344 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq"] Nov 04 01:08:56.437631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437583 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.437631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437632 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.437917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437662 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 04 01:08:56.437917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437756 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.437917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437800 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bed0a38-3d09-483e-8c64-2bc858586a46-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.437917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437828 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4z9n\" (UniqueName: \"kubernetes.io/projected/4bed0a38-3d09-483e-8c64-2bc858586a46-kube-api-access-d4z9n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.437917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.437905 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538653 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538716 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bed0a38-3d09-483e-8c64-2bc858586a46-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538754 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4z9n\" (UniqueName: \"kubernetes.io/projected/4bed0a38-3d09-483e-8c64-2bc858586a46-kube-api-access-d4z9n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538794 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538845 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.538906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.538865 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.539216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.539164 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.539216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.539193 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.539703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.539679 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.542531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.542500 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bed0a38-3d09-483e-8c64-2bc858586a46-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.555052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.555011 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bed0a38-3d09-483e-8c64-2bc858586a46-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.594553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.594469 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4z9n\" (UniqueName: \"kubernetes.io/projected/4bed0a38-3d09-483e-8c64-2bc858586a46-kube-api-access-d4z9n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8vwvq\" (UID: \"4bed0a38-3d09-483e-8c64-2bc858586a46\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:56.662159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:56.662116 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:57.042718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:57.042692 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq"] Nov 04 01:08:57.043976 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:08:57.043948 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bed0a38_3d09_483e_8c64_2bc858586a46.slice/crio-59fdc5fc90a8eebda54404815d11c589c5ca05d1e1917c7c81e1709ddee4ea10 WatchSource:0}: Error finding container 59fdc5fc90a8eebda54404815d11c589c5ca05d1e1917c7c81e1709ddee4ea10: Status 404 returned error can't find the container with id 59fdc5fc90a8eebda54404815d11c589c5ca05d1e1917c7c81e1709ddee4ea10 Nov 04 01:08:57.895789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:57.895758 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" event={"ID":"4bed0a38-3d09-483e-8c64-2bc858586a46","Type":"ContainerStarted","Data":"19716a239c08a12b7a13551782504222d241efc9e3a6265040df89aeaad7c1a3"} Nov 04 01:08:57.896171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:57.895808 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" event={"ID":"4bed0a38-3d09-483e-8c64-2bc858586a46","Type":"ContainerStarted","Data":"59fdc5fc90a8eebda54404815d11c589c5ca05d1e1917c7c81e1709ddee4ea10"} Nov 04 01:08:58.007541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:58.007452 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" podStartSLOduration=1.361321749 podStartE2EDuration="2.007438831s" totalImagesPullingTime="646.117082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:08:58.00580419 +0000 UTC m=+706.764466350" watchObservedRunningTime="2026-11-04 01:08:58.007438831 +0000 UTC m=+706.766100992" Nov 04 01:08:58.662951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:58.662916 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:08:58.818198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:58.818163 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:58.830349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:58.830326 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:08:58.912087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:08:58.912064 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" Nov 04 01:09:06.663124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:06.663087 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:09:06.675681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:06.675652 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:09:06.961602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:06.961522 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" Nov 04 01:09:08.994093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:08.994058 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:09:09.002734 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.002702 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061142 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061198 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm4z\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061219 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061289 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061347 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061510 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061557 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061649 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.061705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.061684 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.110486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.110448 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:09:09.162712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162674 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm4z\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.162885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162719 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.162885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162735 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.162885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162754 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.162885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162829 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.162885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162870 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.162910 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163099 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163162 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163268 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163367 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163543 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163634 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.163808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.163788 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.165131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.165108 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.165393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.165367 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.223465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.223433 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm4z\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.227394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.227371 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-l62hz\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.316794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.316766 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:09.742898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.742871 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:09:09.744759 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:09.744731 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4087ecc_3046_458d_a48a_9bce2b8ace63.slice/crio-4744dd3ec05cb76c28a93d151c0f8e57461976b7d3b5d0d9bc71bd93ec2dde61 WatchSource:0}: Error finding container 4744dd3ec05cb76c28a93d151c0f8e57461976b7d3b5d0d9bc71bd93ec2dde61: Status 404 returned error can't find the container with id 4744dd3ec05cb76c28a93d151c0f8e57461976b7d3b5d0d9bc71bd93ec2dde61 Nov 04 01:09:09.747119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.747083 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:09:09.747191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.747165 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:09:09.747229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.747209 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:09:09.970770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.970731 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" event={"ID":"c4087ecc-3046-458d-a48a-9bce2b8ace63","Type":"ContainerStarted","Data":"fd6cef264d96e8ceb5c06682ed8cb9e8ce7fdc132cce16f397d82668c6358eed"} Nov 04 01:09:09.970950 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:09.970796 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" event={"ID":"c4087ecc-3046-458d-a48a-9bce2b8ace63","Type":"ContainerStarted","Data":"4744dd3ec05cb76c28a93d151c0f8e57461976b7d3b5d0d9bc71bd93ec2dde61"} Nov 04 01:09:10.092621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:10.092565 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podStartSLOduration=2.092549281 podStartE2EDuration="2.092549281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:10.091247945 +0000 UTC m=+718.849910106" watchObservedRunningTime="2026-11-04 01:09:10.092549281 +0000 UTC m=+718.851211443" Nov 04 01:09:10.318007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:10.317950 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:10.319553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:10.319528 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:10.319689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:10.319575 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:11.317259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:11.317227 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:11.317612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:11.317279 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:12.317874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:12.317837 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:12.318381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:12.317888 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:13.318151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:13.318116 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:13.318525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:13.318172 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:13.864784 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:09:13.863174 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:09:14.317867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:14.317831 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:14.318035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:14.317895 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:15.317968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:15.317933 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:15.318446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:15.318014 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:16.317345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:16.317311 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:09:16.317522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:16.317366 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 04 01:09:18.318520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:18.318479 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:09:18.319086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:18.318540 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:09:19.360176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:19.360122 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-l62hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:09:19.360699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:19.360193 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:09:19.738859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:19.738785 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:19.739371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:19.739344 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:19.740088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:19.740069 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:09:39.137017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.136966 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59994988cf-d8rwg"] Nov 04 01:09:39.140809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.140788 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.177249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.177216 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68rk\" (UniqueName: \"kubernetes.io/projected/de8185b1-5c0e-420c-9176-c907147bb322-kube-api-access-h68rk\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.177406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.177384 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de8185b1-5c0e-420c-9176-c907147bb322-tls-cert\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.193847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.193806 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59994988cf-d8rwg"] Nov 04 01:09:39.278769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.278733 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68rk\" (UniqueName: \"kubernetes.io/projected/de8185b1-5c0e-420c-9176-c907147bb322-kube-api-access-h68rk\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.278924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.278844 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de8185b1-5c0e-420c-9176-c907147bb322-tls-cert\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.281335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.281309 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de8185b1-5c0e-420c-9176-c907147bb322-tls-cert\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.335174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.335140 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68rk\" (UniqueName: \"kubernetes.io/projected/de8185b1-5c0e-420c-9176-c907147bb322-kube-api-access-h68rk\") pod \"authorino-59994988cf-d8rwg\" (UID: \"de8185b1-5c0e-420c-9176-c907147bb322\") " pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.351768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.351714 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:09:39.359932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.359880 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.402755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.402679 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:09:39.411349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.411317 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-zdtlk\"" Nov 04 01:09:39.411349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.411336 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 04 01:09:39.411349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.411324 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 04 01:09:39.452829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.452796 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59994988cf-d8rwg" Nov 04 01:09:39.480024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.479996 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59qxf\" (UniqueName: \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.480154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.480078 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.580735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.580701 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.580902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.580783 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59qxf\" (UniqueName: \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.583506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.583471 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.634571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.634537 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59qxf\" (UniqueName: \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\") pod \"ai-gateway-controller-86c7698f44-ml29z\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.679167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.679103 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:39.826135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:39.826105 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59994988cf-d8rwg"] Nov 04 01:09:39.827472 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:39.827442 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8185b1_5c0e_420c_9176_c907147bb322.slice/crio-8713489f443bbbedb0730fdea7a82f20fc2ad3e5993a9be2a3ef2c65ca47fd01 WatchSource:0}: Error finding container 8713489f443bbbedb0730fdea7a82f20fc2ad3e5993a9be2a3ef2c65ca47fd01: Status 404 returned error can't find the container with id 8713489f443bbbedb0730fdea7a82f20fc2ad3e5993a9be2a3ef2c65ca47fd01 Nov 04 01:09:40.051853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:40.051828 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:09:40.053417 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:40.053388 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c5aa39_0f20_4e9a_9d6b_f823cffeeab1.slice/crio-40fe860aa06a46ff721d521b7c4a71ad80652e4b20775f957de933b378f119fe WatchSource:0}: Error finding container 40fe860aa06a46ff721d521b7c4a71ad80652e4b20775f957de933b378f119fe: Status 404 returned error can't find the container with id 40fe860aa06a46ff721d521b7c4a71ad80652e4b20775f957de933b378f119fe Nov 04 01:09:40.131994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:40.131944 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59994988cf-d8rwg" event={"ID":"de8185b1-5c0e-420c-9176-c907147bb322","Type":"ContainerStarted","Data":"8713489f443bbbedb0730fdea7a82f20fc2ad3e5993a9be2a3ef2c65ca47fd01"} Nov 04 01:09:40.133218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:40.133193 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" event={"ID":"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1","Type":"ContainerStarted","Data":"40fe860aa06a46ff721d521b7c4a71ad80652e4b20775f957de933b378f119fe"} Nov 04 01:09:42.145347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.145301 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59994988cf-d8rwg" event={"ID":"de8185b1-5c0e-420c-9176-c907147bb322","Type":"ContainerStarted","Data":"7d272c43fbf17a52a3f9f5d99d391c56d400147cbbfaf619a4c9662fc510f2d7"} Nov 04 01:09:42.253849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.253797 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59994988cf-d8rwg" podStartSLOduration=1.485810192 podStartE2EDuration="3.25378352s" totalImagesPullingTime="1.767973328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:42.252546301 +0000 UTC m=+751.011208462" watchObservedRunningTime="2026-11-04 01:09:42.25378352 +0000 UTC m=+751.012445682" Nov 04 01:09:42.302832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.302795 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:09:42.303106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.303020 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69b47595b-8djwq" podUID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" containerName="authorino" containerID="cri-o://7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd" gracePeriod=30 Nov 04 01:09:42.560015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.558577 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:09:42.609264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.609234 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\" (UniqueName: \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\") pod \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " Nov 04 01:09:42.609452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.609425 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\" (UniqueName: \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\") pod \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\" (UID: \"6390c4c8-be4c-4a1e-854b-8471112fc3a5\") " Nov 04 01:09:42.611443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.611409 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99" pod "6390c4c8-be4c-4a1e-854b-8471112fc3a5" (UID: "6390c4c8-be4c-4a1e-854b-8471112fc3a5"). InnerVolumeSpecName "kube-api-access-dzd99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:09:42.623680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.623653 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert" pod "6390c4c8-be4c-4a1e-854b-8471112fc3a5" (UID: "6390c4c8-be4c-4a1e-854b-8471112fc3a5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:09:42.710336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.710256 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzd99\" (UniqueName: \"kubernetes.io/projected/6390c4c8-be4c-4a1e-854b-8471112fc3a5-kube-api-access-dzd99\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:42.710336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:42.710286 4486 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6390c4c8-be4c-4a1e-854b-8471112fc3a5-tls-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:09:43.153587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.153548 4486 generic.go:356] "Generic (PLEG): container finished" podID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" containerID="7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd" exitCode=0 Nov 04 01:09:43.153587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.153595 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b47595b-8djwq" Nov 04 01:09:43.154161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.153644 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b47595b-8djwq" event={"ID":"6390c4c8-be4c-4a1e-854b-8471112fc3a5","Type":"ContainerDied","Data":"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd"} Nov 04 01:09:43.154161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.153692 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b47595b-8djwq" event={"ID":"6390c4c8-be4c-4a1e-854b-8471112fc3a5","Type":"ContainerDied","Data":"cf3ba9db5153036a5cde1f4f50e49f5caf778a2320e91f19105431e3bd4cd674"} Nov 04 01:09:43.154161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.153714 4486 scope.go:122] "RemoveContainer" containerID="7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd" Nov 04 01:09:43.165245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.165223 4486 scope.go:122] "RemoveContainer" containerID="7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd" Nov 04 01:09:43.165498 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:09:43.165479 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd\": container with ID starting with 7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd not found: ID does not exist" containerID="7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd" Nov 04 01:09:43.165568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.165507 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd"} err="failed to get container status \"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd\": rpc error: code = NotFound desc = could not find container \"7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd\": container with ID starting with 7abea30850c467d286a0a42d1d51868bfdc54ca809a0d807a8fbbec666e993bd not found: ID does not exist" Nov 04 01:09:43.310604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.310567 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:09:43.314954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.314929 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69b47595b-8djwq"] Nov 04 01:09:43.347194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:43.347160 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" path="/var/lib/kubelet/pods/6390c4c8-be4c-4a1e-854b-8471112fc3a5/volumes" Nov 04 01:09:43.834395 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:09:43.834360 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:09:46.177189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:46.177147 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" event={"ID":"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1","Type":"ContainerStarted","Data":"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167"} Nov 04 01:09:46.177576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:46.177216 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:46.288933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:46.288881 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" podStartSLOduration=1.57679071 podStartE2EDuration="7.288866618s" totalImagesPullingTime="5.712075908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:46.285641832 +0000 UTC m=+755.044303994" watchObservedRunningTime="2026-11-04 01:09:46.288866618 +0000 UTC m=+755.047528778" Nov 04 01:09:48.943925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:48.943889 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2"] Nov 04 01:09:48.944538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:48.944517 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" containerName="authorino" Nov 04 01:09:48.944621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:48.944540 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" containerName="authorino" Nov 04 01:09:48.944696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:48.944682 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6390c4c8-be4c-4a1e-854b-8471112fc3a5" containerName="authorino" Nov 04 01:09:48.947558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:48.947537 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.045057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.044873 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 04 01:09:49.045057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.044954 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j2wf8\"" Nov 04 01:09:49.048848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.048816 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2"] Nov 04 01:09:49.080355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.080308 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdv55\" (UniqueName: \"kubernetes.io/projected/ab5b3264-5961-4fc2-be5d-5c68426bb887-kube-api-access-mdv55\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.080532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.080463 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5b3264-5961-4fc2-be5d-5c68426bb887-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.095061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.095030 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xzdcs"] Nov 04 01:09:49.098844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.098810 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.145768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.145738 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xzdcs"] Nov 04 01:09:49.181463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.181423 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hslcz\" (UniqueName: \"kubernetes.io/projected/15448f77-e10f-485b-903a-2c96a15f4460-kube-api-access-hslcz\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.181627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.181480 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5b3264-5961-4fc2-be5d-5c68426bb887-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.181687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.181622 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15448f77-e10f-485b-903a-2c96a15f4460-plugins-config-volume\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.181749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.181728 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdv55\" (UniqueName: \"kubernetes.io/projected/ab5b3264-5961-4fc2-be5d-5c68426bb887-kube-api-access-mdv55\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.182353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.182332 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5b3264-5961-4fc2-be5d-5c68426bb887-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.235851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.235771 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdv55\" (UniqueName: \"kubernetes.io/projected/ab5b3264-5961-4fc2-be5d-5c68426bb887-kube-api-access-mdv55\") pod \"payload-pre-processing-7b98bb678c-v7hk2\" (UID: \"ab5b3264-5961-4fc2-be5d-5c68426bb887\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.265693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.265659 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:49.283231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.283201 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15448f77-e10f-485b-903a-2c96a15f4460-plugins-config-volume\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.283348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.283299 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hslcz\" (UniqueName: \"kubernetes.io/projected/15448f77-e10f-485b-903a-2c96a15f4460-kube-api-access-hslcz\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.284125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.284099 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15448f77-e10f-485b-903a-2c96a15f4460-plugins-config-volume\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.337895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.337860 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hslcz\" (UniqueName: \"kubernetes.io/projected/15448f77-e10f-485b-903a-2c96a15f4460-kube-api-access-hslcz\") pod \"payload-processing-7bb9489676-xzdcs\" (UID: \"15448f77-e10f-485b-903a-2c96a15f4460\") " pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.414520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.414490 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:49.631133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.631103 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2"] Nov 04 01:09:49.632071 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:49.632043 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5b3264_5961_4fc2_be5d_5c68426bb887.slice/crio-02b65d48c09aa7e5ddc5b38e50abcfb5ed3998e839c4e30ac5368a86a4fccb6e WatchSource:0}: Error finding container 02b65d48c09aa7e5ddc5b38e50abcfb5ed3998e839c4e30ac5368a86a4fccb6e: Status 404 returned error can't find the container with id 02b65d48c09aa7e5ddc5b38e50abcfb5ed3998e839c4e30ac5368a86a4fccb6e Nov 04 01:09:49.779554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:49.779530 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xzdcs"] Nov 04 01:09:49.781266 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:49.781234 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15448f77_e10f_485b_903a_2c96a15f4460.slice/crio-ae17c5e7448383498bb20f65a8b71481e8e9758970ce99a6bdfef080a2b64cb6 WatchSource:0}: Error finding container ae17c5e7448383498bb20f65a8b71481e8e9758970ce99a6bdfef080a2b64cb6: Status 404 returned error can't find the container with id ae17c5e7448383498bb20f65a8b71481e8e9758970ce99a6bdfef080a2b64cb6 Nov 04 01:09:50.200744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:50.200704 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" event={"ID":"15448f77-e10f-485b-903a-2c96a15f4460","Type":"ContainerStarted","Data":"ae17c5e7448383498bb20f65a8b71481e8e9758970ce99a6bdfef080a2b64cb6"} Nov 04 01:09:50.201771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:50.201745 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" event={"ID":"ab5b3264-5961-4fc2-be5d-5c68426bb887","Type":"ContainerStarted","Data":"02b65d48c09aa7e5ddc5b38e50abcfb5ed3998e839c4e30ac5368a86a4fccb6e"} Nov 04 01:09:53.232163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.232049 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" event={"ID":"ab5b3264-5961-4fc2-be5d-5c68426bb887","Type":"ContainerStarted","Data":"dc73a9d288545906a8612097268148fe0bd6251fc1b5c514ea8d6fb5b3019ad1"} Nov 04 01:09:53.232163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.232157 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:09:53.234232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.234201 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" event={"ID":"15448f77-e10f-485b-903a-2c96a15f4460","Type":"ContainerStarted","Data":"82125d76e480696c27ab5d8d32e337c459dcfbc1cde96d7011a348b3d033e903"} Nov 04 01:09:53.234363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.234287 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:09:53.342597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.342547 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" podStartSLOduration=2.094182006 podStartE2EDuration="5.342532764s" totalImagesPullingTime="3.248350758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:53.339947794 +0000 UTC m=+762.098609957" watchObservedRunningTime="2026-11-04 01:09:53.342532764 +0000 UTC m=+762.101194924" Nov 04 01:09:53.448217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:53.448167 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" podStartSLOduration=1.1733512130000001 podStartE2EDuration="4.44815212s" totalImagesPullingTime="3.274800907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:53.446855094 +0000 UTC m=+762.205517255" watchObservedRunningTime="2026-11-04 01:09:53.44815212 +0000 UTC m=+762.206814282" Nov 04 01:09:57.185591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.185562 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:09:57.523464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.523383 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5cc77699f-8pjtq"] Nov 04 01:09:57.526926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.526910 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.581912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.581885 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5cc77699f-8pjtq"] Nov 04 01:09:57.666893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.666861 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sgp\" (UniqueName: \"kubernetes.io/projected/9ab844e2-731c-417c-8630-09f10b5f05b3-kube-api-access-t4sgp\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.667053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.666904 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab844e2-731c-417c-8630-09f10b5f05b3-metrics-certs\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.768426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.768390 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sgp\" (UniqueName: \"kubernetes.io/projected/9ab844e2-731c-417c-8630-09f10b5f05b3-kube-api-access-t4sgp\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.768574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.768433 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab844e2-731c-417c-8630-09f10b5f05b3-metrics-certs\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.770786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.770763 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ab844e2-731c-417c-8630-09f10b5f05b3-metrics-certs\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.823030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.823006 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sgp\" (UniqueName: \"kubernetes.io/projected/9ab844e2-731c-417c-8630-09f10b5f05b3-kube-api-access-t4sgp\") pod \"ai-gateway-controller-5cc77699f-8pjtq\" (UID: \"9ab844e2-731c-417c-8630-09f10b5f05b3\") " pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:57.841718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:57.841698 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:58.221665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:58.221635 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5cc77699f-8pjtq"] Nov 04 01:09:58.222083 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:09:58.221810 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab844e2_731c_417c_8630_09f10b5f05b3.slice/crio-5313afbf55dd874bc00e85d77e4891d49c0f11fdecde6f34e0fdf4ba3534645d WatchSource:0}: Error finding container 5313afbf55dd874bc00e85d77e4891d49c0f11fdecde6f34e0fdf4ba3534645d: Status 404 returned error can't find the container with id 5313afbf55dd874bc00e85d77e4891d49c0f11fdecde6f34e0fdf4ba3534645d Nov 04 01:09:58.260584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:58.260555 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" event={"ID":"9ab844e2-731c-417c-8630-09f10b5f05b3","Type":"ContainerStarted","Data":"5313afbf55dd874bc00e85d77e4891d49c0f11fdecde6f34e0fdf4ba3534645d"} Nov 04 01:09:59.268971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:59.268938 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" event={"ID":"9ab844e2-731c-417c-8630-09f10b5f05b3","Type":"ContainerStarted","Data":"cd59df3aa6e2c9ea48b0b87380bdf87c59e718c1f9835144835af611475bd34b"} Nov 04 01:09:59.269336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:59.269114 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:09:59.379199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:09:59.379153 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" podStartSLOduration=1.424529458 podStartE2EDuration="2.3791389s" totalImagesPullingTime="954.609442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:09:59.37708689 +0000 UTC m=+768.135749052" watchObservedRunningTime="2026-11-04 01:09:59.3791389 +0000 UTC m=+768.137801062" Nov 04 01:10:04.243053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:04.243023 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-xzdcs" Nov 04 01:10:04.243469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:04.243099 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7hk2" Nov 04 01:10:10.277411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.277370 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5cc77699f-8pjtq" Nov 04 01:10:10.418758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.418719 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:10:10.418975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.418951 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" podUID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" containerName="manager" containerID="cri-o://941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167" gracePeriod=30 Nov 04 01:10:10.672478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.672455 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:10:10.788101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.788038 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\" (UniqueName: \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\") pod \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " Nov 04 01:10:10.788101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.788074 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\") pod \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\" (UID: \"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1\") " Nov 04 01:10:10.790103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.790073 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf" pod "93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" (UID: "93c5aa39-0f20-4e9a-9d6b-f823cffeeab1"). InnerVolumeSpecName "kube-api-access-59qxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:10:10.790252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.790231 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs" pod "93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" (UID: "93c5aa39-0f20-4e9a-9d6b-f823cffeeab1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:10:10.889183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.889154 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59qxf\" (UniqueName: \"kubernetes.io/projected/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-kube-api-access-59qxf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:10.889183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:10.889178 4486 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1-metrics-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:10:11.340207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.340173 4486 generic.go:356] "Generic (PLEG): container finished" podID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" containerID="941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167" exitCode=0 Nov 04 01:10:11.340642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.340229 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" Nov 04 01:10:11.340642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.340261 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" event={"ID":"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1","Type":"ContainerDied","Data":"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167"} Nov 04 01:10:11.340642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.340340 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-ml29z" event={"ID":"93c5aa39-0f20-4e9a-9d6b-f823cffeeab1","Type":"ContainerDied","Data":"40fe860aa06a46ff721d521b7c4a71ad80652e4b20775f957de933b378f119fe"} Nov 04 01:10:11.340642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.340368 4486 scope.go:122] "RemoveContainer" containerID="941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167" Nov 04 01:10:11.350641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.350563 4486 scope.go:122] "RemoveContainer" containerID="941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167" Nov 04 01:10:11.351039 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:10:11.351012 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167\": container with ID starting with 941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167 not found: ID does not exist" containerID="941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167" Nov 04 01:10:11.351125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.351047 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167"} err="failed to get container status \"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167\": rpc error: code = NotFound desc = could not find container \"941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167\": container with ID starting with 941903e78f835696e5412fd6a277f390925d32af5dfd465c5ce301a31152a167 not found: ID does not exist" Nov 04 01:10:11.499029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.498997 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:10:11.501825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:11.501801 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-ml29z"] Nov 04 01:10:13.347883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:13.347846 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" path="/var/lib/kubelet/pods/93c5aa39-0f20-4e9a-9d6b-f823cffeeab1/volumes" Nov 04 01:10:13.833373 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:10:13.833339 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:10:43.468837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.468798 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:10:43.469366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.469302 4486 cpu_manager.go:414] "containerMap: removing container" podUID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" containerName="manager" Nov 04 01:10:43.469366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.469319 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" containerName="manager" Nov 04 01:10:43.469453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.469421 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c5aa39-0f20-4e9a-9d6b-f823cffeeab1" containerName="manager" Nov 04 01:10:43.473532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.473512 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.526090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.526052 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e210ef-openshift-default-dockercfg-qv9pj\"" Nov 04 01:10:43.576061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.576025 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:10:43.596155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596119 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596166 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596256 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596290 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596403 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596438 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596460 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvxv\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596502 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.596561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.596550 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697146 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697197 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697222 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697246 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697390 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697449 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697482 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvxv\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.697523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697513 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697544 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.697653 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.698243 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.698333 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.698363 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.698536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.698515 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.699946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.699924 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.700143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.700117 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.751874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.751818 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.751976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.751957 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvxv\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.785924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:43.785897 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:43.837287 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:10:43.837215 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:10:44.120412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.120370 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:10:44.125010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.124970 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.183004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.181781 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2fd27d-openshift-default-dockercfg-bkzjb\"" Nov 04 01:10:44.193729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.193462 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203671 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203739 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203840 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203885 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203913 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.203963 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.204031 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.204078 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrb7c\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.208224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.204170 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.241500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.241472 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:10:44.243722 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:10:44.243115 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0f9407_291d_4c95_a83d_f4b5f88fce9a.slice/crio-7f1bfc6da6d36c0e6bbf82c49bfe1a7659c6ad89a686b09d197b3dc299af5a75 WatchSource:0}: Error finding container 7f1bfc6da6d36c0e6bbf82c49bfe1a7659c6ad89a686b09d197b3dc299af5a75: Status 404 returned error can't find the container with id 7f1bfc6da6d36c0e6bbf82c49bfe1a7659c6ad89a686b09d197b3dc299af5a75 Nov 04 01:10:44.245194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.245158 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.245296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.245234 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.245296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.245277 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.305376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305349 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305395 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305515 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305648 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305687 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305715 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305753 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305807 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305839 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrb7c\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.305972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.305899 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.306057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.306028 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.306198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.306173 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.306331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.306301 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.306586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.306564 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.308249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.308223 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.308347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.308304 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.359935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.359898 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.360194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.360173 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrb7c\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.439050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.438961 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:44.526475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.526436 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerStarted","Data":"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8"} Nov 04 01:10:44.526855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.526507 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerStarted","Data":"7f1bfc6da6d36c0e6bbf82c49bfe1a7659c6ad89a686b09d197b3dc299af5a75"} Nov 04 01:10:44.638866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.638815 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podStartSLOduration=1.638801272 podStartE2EDuration="1.638801272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:44.637561623 +0000 UTC m=+813.396223785" watchObservedRunningTime="2026-11-04 01:10:44.638801272 +0000 UTC m=+813.397463432" Nov 04 01:10:44.787131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.787057 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:44.794650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.794622 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:10:44.799707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.799688 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.867635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.860151 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:10:44.867635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.862408 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:10:44.867635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.867618 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.867823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.867669 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.867823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.867698 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:44.889035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.889015 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c563ee-openshift-default-dockercfg-5mps8\"" Nov 04 01:10:44.914842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.914813 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.914871 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.914951 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6r2k\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.914993 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.915049 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.915305 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.915378 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.915410 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:44.915498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:44.915458 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.016720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016684 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.016720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016731 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016757 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016805 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016857 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016922 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6r2k\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.016948 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.017009 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.017040 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.017717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.017431 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.018458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.018065 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.018458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.018410 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.019233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.019194 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.019384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.019367 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.020757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.020567 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.020757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.020597 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.072342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.072311 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.072598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.072576 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6r2k\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.113329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.113297 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:45.485038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.484938 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:10:45.489840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.489815 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.534650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.534615 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerStarted","Data":"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb"} Nov 04 01:10:45.535060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.534674 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerStarted","Data":"ab28cc5cd2967396f4011d2718219bb0f2e8fcd0953b583a0df7ea5b53ccf0ab"} Nov 04 01:10:45.545351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.545313 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:10:45.547344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.547266 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e65b23-openshift-default-dockercfg-h6b8n\"" Nov 04 01:10:45.553438 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:10:45.553412 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be6c0dd_2e33_4dc0_8e81_e0684a23fd25.slice/crio-037c99186b47e366bf88656c91d7e3a7163300f308454732ba38a05b70fe81b8 WatchSource:0}: Error finding container 037c99186b47e366bf88656c91d7e3a7163300f308454732ba38a05b70fe81b8: Status 404 returned error can't find the container with id 037c99186b47e366bf88656c91d7e3a7163300f308454732ba38a05b70fe81b8 Nov 04 01:10:45.555607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.555579 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:45.555690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.555638 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:45.555690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.555666 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:45.569653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.569623 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:10:45.622868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.622835 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.622974 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623215 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623283 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623384 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623422 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623518 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623563 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfbb\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.623680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.623661 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725075 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725128 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfbb\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725168 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725263 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725316 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725353 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725390 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725481 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725520 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725829 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725872 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.725974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.725949 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.726369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.726350 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.728349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.728326 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.729620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.729596 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.779475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.779407 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfbb\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.779716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.779688 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.793190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.793163 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:45.803738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.803707 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:45.922398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:45.922337 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podStartSLOduration=1.922314814 podStartE2EDuration="1.922314814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:45.721447104 +0000 UTC m=+814.480109266" watchObservedRunningTime="2026-11-04 01:10:45.922314814 +0000 UTC m=+814.680976977" Nov 04 01:10:46.191300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.191271 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:10:46.192288 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:10:46.192264 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee6b30f_f5ca_4118_8132_d4328116365b.slice/crio-a154e8b5a9b2404ac4863a09eecc0903ea56e5dd04302f957064619fb8cb7851 WatchSource:0}: Error finding container a154e8b5a9b2404ac4863a09eecc0903ea56e5dd04302f957064619fb8cb7851: Status 404 returned error can't find the container with id a154e8b5a9b2404ac4863a09eecc0903ea56e5dd04302f957064619fb8cb7851 Nov 04 01:10:46.194651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.194621 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:46.194718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.194684 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:46.194751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.194729 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:10:46.439860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.439772 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:46.445172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.445148 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:46.544821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.544787 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerStarted","Data":"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1"} Nov 04 01:10:46.545316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.545154 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerStarted","Data":"037c99186b47e366bf88656c91d7e3a7163300f308454732ba38a05b70fe81b8"} Nov 04 01:10:46.547698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.547657 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerStarted","Data":"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc"} Nov 04 01:10:46.548549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.548436 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:46.548549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.548498 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:46.548549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.548521 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerStarted","Data":"a154e8b5a9b2404ac4863a09eecc0903ea56e5dd04302f957064619fb8cb7851"} Nov 04 01:10:46.549527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.549507 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:10:46.549614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.549559 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:10:46.668864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.668800 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podStartSLOduration=2.668783501 podStartE2EDuration="2.668783501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:46.665170416 +0000 UTC m=+815.423832591" watchObservedRunningTime="2026-11-04 01:10:46.668783501 +0000 UTC m=+815.427445664" Nov 04 01:10:46.805157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.805081 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:46.998092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:46.998041 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podStartSLOduration=1.9980254689999999 podStartE2EDuration="1.998025469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:46.996054748 +0000 UTC m=+815.754716910" watchObservedRunningTime="2026-11-04 01:10:46.998025469 +0000 UTC m=+815.756687626" Nov 04 01:10:47.125152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:47.125124 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:47.135167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:47.135137 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:47.553112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:47.553025 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:47.553896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:47.553876 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:10:47.809543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:47.809467 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:48.557924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:48.557886 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:48.558464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:48.558446 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:10:58.145898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.145865 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:10:58.149829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.149808 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.157920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.157642 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44hx\" (UniqueName: \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.157920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.157692 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.204586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.204559 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:10:58.247172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.247139 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2fd27d\"" Nov 04 01:10:58.247666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.247642 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2fd27d-dockercfg-b526g\"" Nov 04 01:10:58.258540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.258493 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44hx\" (UniqueName: \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.258695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.258570 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.259730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.259700 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.297730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.297696 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:10:58.301608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.301588 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.313176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.313152 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44hx\" (UniqueName: \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\") pod \"payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.339945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.339919 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:10:58.359581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.359555 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xh88\" (UniqueName: \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.359690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.359591 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.460431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.460357 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xh88\" (UniqueName: \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.460566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.460440 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.461187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.461166 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.464050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.464026 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:58.515572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.515542 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xh88\" (UniqueName: \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\") pod \"payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.615017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.614972 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:58.833170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.833143 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:10:58.833968 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:10:58.833933 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd726207_bfe1_42c9_a2ae_8b90e9084082.slice/crio-d84ae038368cc44a7eca9435cd94186ca0b84fce8ba4d45b02d051b42c1d7beb WatchSource:0}: Error finding container d84ae038368cc44a7eca9435cd94186ca0b84fce8ba4d45b02d051b42c1d7beb: Status 404 returned error can't find the container with id d84ae038368cc44a7eca9435cd94186ca0b84fce8ba4d45b02d051b42c1d7beb Nov 04 01:10:58.988771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:58.988741 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:10:58.989798 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:10:58.989771 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4846dc50_4398_4e49_8d6b_725a49dc0007.slice/crio-c5ce6da38ced974bf010a3371ab2d462972e2cae8549e5c878dc4e5d35834a91 WatchSource:0}: Error finding container c5ce6da38ced974bf010a3371ab2d462972e2cae8549e5c878dc4e5d35834a91: Status 404 returned error can't find the container with id c5ce6da38ced974bf010a3371ab2d462972e2cae8549e5c878dc4e5d35834a91 Nov 04 01:10:59.632430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.632387 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" event={"ID":"dd726207-bfe1-42c9-a2ae-8b90e9084082","Type":"ContainerStarted","Data":"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c"} Nov 04 01:10:59.632898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.632450 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" event={"ID":"dd726207-bfe1-42c9-a2ae-8b90e9084082","Type":"ContainerStarted","Data":"d84ae038368cc44a7eca9435cd94186ca0b84fce8ba4d45b02d051b42c1d7beb"} Nov 04 01:10:59.632898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.632481 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:10:59.634079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.634055 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" event={"ID":"4846dc50-4398-4e49-8d6b-725a49dc0007","Type":"ContainerStarted","Data":"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d"} Nov 04 01:10:59.634192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.634103 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" event={"ID":"4846dc50-4398-4e49-8d6b-725a49dc0007","Type":"ContainerStarted","Data":"c5ce6da38ced974bf010a3371ab2d462972e2cae8549e5c878dc4e5d35834a91"} Nov 04 01:10:59.634192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.634132 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:10:59.744368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.744317 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" podStartSLOduration=1.744303644 podStartE2EDuration="1.744303644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:59.742528011 +0000 UTC m=+828.501190173" watchObservedRunningTime="2026-11-04 01:10:59.744303644 +0000 UTC m=+828.502965805" Nov 04 01:10:59.852248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:10:59.852193 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" podStartSLOduration=1.852178174 podStartE2EDuration="1.852178174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:10:59.85152414 +0000 UTC m=+828.610186304" watchObservedRunningTime="2026-11-04 01:10:59.852178174 +0000 UTC m=+828.610840335" Nov 04 01:11:00.674229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.674198 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:11:00.678092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.678071 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.728820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.728783 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e993d7\"" Nov 04 01:11:00.732819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.732790 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:11:00.786425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.786390 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e993d7-dockercfg-gfv6p\"" Nov 04 01:11:00.788184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.787345 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfbf\" (UniqueName: \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.788184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.787408 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.817718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.817683 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:11:00.821449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.821432 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:00.875365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.875336 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:11:00.889135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.889107 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfbf\" (UniqueName: \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.889292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.889151 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.889292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.889179 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:00.889292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.889229 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblxz\" (UniqueName: \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:00.889841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.889820 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.943463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.943393 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfbf\" (UniqueName: \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\") pod \"payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.990529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.990492 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:00.990709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.990545 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblxz\" (UniqueName: \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:00.991245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.991213 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:00.991245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:00.991232 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:01.045063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.045005 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblxz\" (UniqueName: \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\") pod \"payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:01.135614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.135574 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:01.365091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.365056 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:11:01.366163 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:01.366130 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e78acf8_cf6b_4ae2_9875_a2a7f3c176e0.slice/crio-6e0e92e3cec22cf5f7618800dfeb3d294e4c9bf4cd94690bff801573d16f1c89 WatchSource:0}: Error finding container 6e0e92e3cec22cf5f7618800dfeb3d294e4c9bf4cd94690bff801573d16f1c89: Status 404 returned error can't find the container with id 6e0e92e3cec22cf5f7618800dfeb3d294e4c9bf4cd94690bff801573d16f1c89 Nov 04 01:11:01.509848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.509821 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:11:01.510354 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:01.510326 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ae019e_b51e_4576_9487_4ebc38c89d1a.slice/crio-e6945417bb20eee3fdb5fbc8f6eed37689ab9a1bc4fd1886da2d025f06538c5e WatchSource:0}: Error finding container e6945417bb20eee3fdb5fbc8f6eed37689ab9a1bc4fd1886da2d025f06538c5e: Status 404 returned error can't find the container with id e6945417bb20eee3fdb5fbc8f6eed37689ab9a1bc4fd1886da2d025f06538c5e Nov 04 01:11:01.656293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.656237 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" event={"ID":"36ae019e-b51e-4576-9487-4ebc38c89d1a","Type":"ContainerStarted","Data":"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe"} Nov 04 01:11:01.656499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.656352 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" event={"ID":"36ae019e-b51e-4576-9487-4ebc38c89d1a","Type":"ContainerStarted","Data":"e6945417bb20eee3fdb5fbc8f6eed37689ab9a1bc4fd1886da2d025f06538c5e"} Nov 04 01:11:01.656499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.656384 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:01.658060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.658032 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" event={"ID":"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0","Type":"ContainerStarted","Data":"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864"} Nov 04 01:11:01.658189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.658084 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" event={"ID":"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0","Type":"ContainerStarted","Data":"6e0e92e3cec22cf5f7618800dfeb3d294e4c9bf4cd94690bff801573d16f1c89"} Nov 04 01:11:01.658189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.658133 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:01.767206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.767095 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" podStartSLOduration=1.767081445 podStartE2EDuration="1.767081445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:01.765474725 +0000 UTC m=+830.524136891" watchObservedRunningTime="2026-11-04 01:11:01.767081445 +0000 UTC m=+830.525743605" Nov 04 01:11:01.879541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:01.879471 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" podStartSLOduration=1.879449094 podStartE2EDuration="1.879449094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:01.875873021 +0000 UTC m=+830.634535164" watchObservedRunningTime="2026-11-04 01:11:01.879449094 +0000 UTC m=+830.638111256" Nov 04 01:11:05.658834 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.658800 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:11:05.663085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.663061 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.714898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.714872 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:11:05.758565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.758533 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-678aaf-dockercfg-z5295\"" Nov 04 01:11:05.761429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.761410 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-678aaf\"" Nov 04 01:11:05.765736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.765705 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.765838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.765756 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstn9\" (UniqueName: \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.798079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.798051 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:11:05.801912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.801895 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.869005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.868267 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.869005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.868359 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wg6\" (UniqueName: \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.869005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.868423 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.869005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.868451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstn9\" (UniqueName: \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.878045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.869584 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.889712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.889680 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:11:05.923769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.923696 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstn9\" (UniqueName: \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\") pod \"payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:05.969072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.969038 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.969242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.969132 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wg6\" (UniqueName: \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.969966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.969943 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:05.976931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:05.976910 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:06.059007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.038754 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wg6\" (UniqueName: \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\") pod \"payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:06.115259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.115234 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:06.372441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.372413 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:11:06.373499 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:06.373471 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026e0e82_97c5_4662_bb88_ec0937b8f5f0.slice/crio-954f9ce68e106e7bc8fc3d9aed5fe6e8698cdadff62d420b828b29f5b052f11b WatchSource:0}: Error finding container 954f9ce68e106e7bc8fc3d9aed5fe6e8698cdadff62d420b828b29f5b052f11b: Status 404 returned error can't find the container with id 954f9ce68e106e7bc8fc3d9aed5fe6e8698cdadff62d420b828b29f5b052f11b Nov 04 01:11:06.511193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.511168 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:11:06.511280 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:06.511242 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5526139_52c9_440d_bc9f_a3bb1fcad224.slice/crio-8b53902499db0892c0a1ed3284d66dc8af74f048ef71d6217580a5887012c11d WatchSource:0}: Error finding container 8b53902499db0892c0a1ed3284d66dc8af74f048ef71d6217580a5887012c11d: Status 404 returned error can't find the container with id 8b53902499db0892c0a1ed3284d66dc8af74f048ef71d6217580a5887012c11d Nov 04 01:11:06.689076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.689036 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" event={"ID":"026e0e82-97c5-4662-bb88-ec0937b8f5f0","Type":"ContainerStarted","Data":"36845891034b56c8cc6956be03240e77485747e5f38fee1746d3fe979fc804bc"} Nov 04 01:11:06.689546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.689101 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" event={"ID":"026e0e82-97c5-4662-bb88-ec0937b8f5f0","Type":"ContainerStarted","Data":"954f9ce68e106e7bc8fc3d9aed5fe6e8698cdadff62d420b828b29f5b052f11b"} Nov 04 01:11:06.689546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.689209 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:06.690695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.690662 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" event={"ID":"c5526139-52c9-440d-bc9f-a3bb1fcad224","Type":"ContainerStarted","Data":"cad94f8d670671aef46871b801c3c964ebff0bc491b001a7294c4dc8c78ee462"} Nov 04 01:11:06.690823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.690702 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" event={"ID":"c5526139-52c9-440d-bc9f-a3bb1fcad224","Type":"ContainerStarted","Data":"8b53902499db0892c0a1ed3284d66dc8af74f048ef71d6217580a5887012c11d"} Nov 04 01:11:06.690823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.690722 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:06.805054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.804932 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" podStartSLOduration=1.80491798 podStartE2EDuration="1.80491798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:06.804086327 +0000 UTC m=+835.562748490" watchObservedRunningTime="2026-11-04 01:11:06.80491798 +0000 UTC m=+835.563580140" Nov 04 01:11:06.914396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:06.914346 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" podStartSLOduration=1.914332336 podStartE2EDuration="1.914332336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:06.912856624 +0000 UTC m=+835.671518822" watchObservedRunningTime="2026-11-04 01:11:06.914332336 +0000 UTC m=+835.672994497" Nov 04 01:11:10.643866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:10.643835 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:11:10.644288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:10.643915 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:11:12.668593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:12.668558 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:11:12.668952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:12.668654 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:11:13.240390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.238188 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:11:13.244028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.244004 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.292434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.292398 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:11:13.293803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.293775 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e65b23-dockercfg-4vcsd\"" Nov 04 01:11:13.361057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.354363 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e65b23\"" Nov 04 01:11:13.363594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.363545 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.363943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.363914 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmnlr\" (UniqueName: \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.465098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.465059 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.465286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.465206 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmnlr\" (UniqueName: \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.466091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.466064 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.520264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.520218 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmnlr\" (UniqueName: \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\") pod \"payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.558994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.558951 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:13.740796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.739454 4486 generic.go:356] "Generic (PLEG): container finished" podID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerID="4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb" exitCode=0 Nov 04 01:11:13.740796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.739615 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerDied","Data":"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb"} Nov 04 01:11:13.740796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.740079 4486 scope.go:122] "RemoveContainer" containerID="4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb" Nov 04 01:11:13.743639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.742808 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:13.743639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.742914 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:13.743639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.742965 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:13.869015 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:11:13.868947 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:11:13.971042 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:13.970217 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:11:13.973310 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:13.972312 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6c003d_ee8f_4685_959b_444edc0d973b.slice/crio-c7593fd6534630293d0fa83731e12d7cc4db9740dcf694836256976c1af69166 WatchSource:0}: Error finding container c7593fd6534630293d0fa83731e12d7cc4db9740dcf694836256976c1af69166: Status 404 returned error can't find the container with id c7593fd6534630293d0fa83731e12d7cc4db9740dcf694836256976c1af69166 Nov 04 01:11:14.763755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:14.763713 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerStarted","Data":"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa"} Nov 04 01:11:14.766016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:14.765974 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" event={"ID":"da6c003d-ee8f-4685-959b-444edc0d973b","Type":"ContainerStarted","Data":"e2d59cceb5db1e02cd83465b19fc11017b8feed40705d8aac76d5916d72340ff"} Nov 04 01:11:14.766169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:14.766033 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" event={"ID":"da6c003d-ee8f-4685-959b-444edc0d973b","Type":"ContainerStarted","Data":"c7593fd6534630293d0fa83731e12d7cc4db9740dcf694836256976c1af69166"} Nov 04 01:11:14.766169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:14.766054 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:15.044058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.043520 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" podStartSLOduration=2.043499764 podStartE2EDuration="2.043499764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:15.038465289 +0000 UTC m=+843.797127451" watchObservedRunningTime="2026-11-04 01:11:15.043499764 +0000 UTC m=+843.802161927" Nov 04 01:11:15.439307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.439276 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:11:15.454368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.454329 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:11:15.454530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.454383 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 01:11:15.956731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.956695 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:11:15.962497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.962476 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:15.996215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:15.996188 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:11:16.002705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.002676 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwhp\" (UniqueName: \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.002838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.002717 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.011842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.011816 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2ee045\"" Nov 04 01:11:16.012161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.012120 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2ee045-dockercfg-m54kt\"" Nov 04 01:11:16.104159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.104113 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwhp\" (UniqueName: \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.104390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.104175 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.104842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.104821 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.158517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.158485 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwhp\" (UniqueName: \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\") pod \"payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.276825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.276754 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.446071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.446034 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:11:16.453061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.446313 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 01:11:16.553541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.551382 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:11:16.673349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.673322 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:11:16.674719 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:11:16.674691 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4039758e_0b4d_460d_b6bd_134b1dbc3764.slice/crio-6f7a5b67d7b2df485b6d10509e03baab699569d14220e260e2d716ebf186a880 WatchSource:0}: Error finding container 6f7a5b67d7b2df485b6d10509e03baab699569d14220e260e2d716ebf186a880: Status 404 returned error can't find the container with id 6f7a5b67d7b2df485b6d10509e03baab699569d14220e260e2d716ebf186a880 Nov 04 01:11:16.782263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.782235 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" event={"ID":"4039758e-0b4d-460d-b6bd-134b1dbc3764","Type":"ContainerStarted","Data":"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac"} Nov 04 01:11:16.782383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.782288 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" event={"ID":"4039758e-0b4d-460d-b6bd-134b1dbc3764","Type":"ContainerStarted","Data":"6f7a5b67d7b2df485b6d10509e03baab699569d14220e260e2d716ebf186a880"} Nov 04 01:11:16.782483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.782465 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:16.908049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:16.907998 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" podStartSLOduration=1.907966551 podStartE2EDuration="1.907966551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:11:16.907673589 +0000 UTC m=+845.666335750" watchObservedRunningTime="2026-11-04 01:11:16.907966551 +0000 UTC m=+845.666628712" Nov 04 01:11:17.447447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:17.447355 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:11:17.453110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:17.447971 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 01:11:17.699583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:17.699507 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:11:17.699746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:17.699593 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:11:18.453242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:18.453192 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:11:18.453829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:18.453798 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 01:11:19.448504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:19.448463 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:11:19.448816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:19.448787 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 04 01:11:20.444219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:20.444189 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:11:20.445448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:20.445420 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:11:25.777620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:25.777534 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:11:27.792506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:27.792473 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:11:32.589209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:32.589161 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:11:32.589627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:32.589248 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:11:33.941755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.941719 4486 generic.go:356] "Generic (PLEG): container finished" podID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerID="d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8" exitCode=0 Nov 04 01:11:33.942198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.941789 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerDied","Data":"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8"} Nov 04 01:11:33.942322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.942291 4486 scope.go:122] "RemoveContainer" containerID="d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8" Nov 04 01:11:33.943708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.943675 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:33.943799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.943745 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:33.943799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:33.943778 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:34.970323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:34.970281 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerStarted","Data":"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff"} Nov 04 01:11:35.786703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:35.786662 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:11:35.786946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:35.786921 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:11:35.787065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:35.786994 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 01:11:36.787607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:36.787552 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:11:36.788182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:36.787632 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 01:11:37.788679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:37.788631 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:11:37.789359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:37.789311 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 01:11:38.786474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:38.786436 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:11:38.786666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:38.786510 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 01:11:39.786705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:39.786657 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:11:39.787163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:39.786718 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 04 01:11:40.791111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:40.791077 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:11:40.792040 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:40.791975 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:11:40.792941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:40.792920 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:11:43.831430 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:11:43.831389 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:11:46.057000 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.056958 4486 generic.go:356] "Generic (PLEG): container finished" podID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerID="fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1" exitCode=0 Nov 04 01:11:46.057422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.057041 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerDied","Data":"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1"} Nov 04 01:11:46.057464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.057435 4486 scope.go:122] "RemoveContainer" containerID="fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1" Nov 04 01:11:46.058741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.058715 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:46.058809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.058782 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:46.058841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:46.058812 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:47.069330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:47.069280 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerStarted","Data":"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1"} Nov 04 01:11:47.114492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:47.114453 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:11:47.115676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:47.115653 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:47.115739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:47.115703 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:47.556207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:47.556096 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:11:48.113757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:48.113715 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:48.114195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:48.113796 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:49.121364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:49.121327 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:49.121941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:49.121907 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:50.114264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:50.114223 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:50.114437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:50.114302 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:51.114628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:51.114590 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:51.115079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:51.114653 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:52.113998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:52.113944 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:11:52.114177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:52.114016 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 04 01:11:53.118023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:53.117965 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:11:53.119223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:53.119201 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:11:58.194231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.194199 4486 generic.go:356] "Generic (PLEG): container finished" podID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerID="e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc" exitCode=0 Nov 04 01:11:58.194639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.194271 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerDied","Data":"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc"} Nov 04 01:11:58.194639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.194610 4486 scope.go:122] "RemoveContainer" containerID="e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc" Nov 04 01:11:58.195920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.195894 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:58.196025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.195970 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:58.196070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:58.196056 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:11:59.202620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:59.202583 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerStarted","Data":"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e"} Nov 04 01:11:59.804571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:59.804528 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:11:59.805900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:59.805871 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:11:59.806045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:11:59.805934 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:12:00.804347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:00.804310 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:12:00.804742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:00.804384 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:12:01.804546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:01.804512 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:12:01.804945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:01.804568 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:12:02.804232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:02.804197 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:12:02.804424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:02.804251 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:12:03.558132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:03.558089 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:12:03.804244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:03.804203 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 04 01:12:03.804406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:03.804265 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 04 01:12:04.807860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:04.807826 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:12:04.808913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:04.808891 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:12:11.426765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:11.426736 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:12:11.433971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:11.433940 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:12:11.442214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:11.442193 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:12:11.455582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:12:11.455550 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:12:13.830176 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:12:13.830131 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:12:43.830666 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:12:43.830628 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:13:13.830301 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:13:13.830259 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:13:17.100638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.100604 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:13:17.103561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.103540 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.161784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.161755 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:13:17.191324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191298 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.191493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191385 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.191571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191496 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.191571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191545 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.191692 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191612 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gms\" (UniqueName: \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.191753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.191700 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.203744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.203720 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"default-dockercfg-cljcx\"" Nov 04 01:13:17.204048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.204032 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"premium-w0-e993d7-kserve-self-signed-certs\"" Nov 04 01:13:17.204108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.204043 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"kube-root-ca.crt\"" Nov 04 01:13:17.204108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.204032 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"openshift-service-ca.crt\"" Nov 04 01:13:17.292799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292764 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292824 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292848 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292865 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gms\" (UniqueName: \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292903 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.292952 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.293338 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.293359 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.293436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.293388 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.295180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.295159 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.295425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.295407 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.369195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.369138 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gms\" (UniqueName: \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\") pod \"premium-w0-e993d7-kserve-8964c87f-ql2zb\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.414137 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.414111 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:17.794441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:17.794410 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:13:17.794645 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:13:17.794620 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8113b2a6_4fed_41d8_be88_d04cac9a8569.slice/crio-c3d80bff11f436afd4b1d631032fa186a0652db8ab929861c8099c1b8b3a6fe0 WatchSource:0}: Error finding container c3d80bff11f436afd4b1d631032fa186a0652db8ab929861c8099c1b8b3a6fe0: Status 404 returned error can't find the container with id c3d80bff11f436afd4b1d631032fa186a0652db8ab929861c8099c1b8b3a6fe0 Nov 04 01:13:18.771229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:18.771195 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" event={"ID":"8113b2a6-4fed-41d8-be88-d04cac9a8569","Type":"ContainerStarted","Data":"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567"} Nov 04 01:13:18.771646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:18.771240 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" event={"ID":"8113b2a6-4fed-41d8-be88-d04cac9a8569","Type":"ContainerStarted","Data":"c3d80bff11f436afd4b1d631032fa186a0652db8ab929861c8099c1b8b3a6fe0"} Nov 04 01:13:18.883063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:18.883007 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podStartSLOduration=1.8829759940000002 podStartE2EDuration="1.882975994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:18.881749049 +0000 UTC m=+967.640411212" watchObservedRunningTime="2026-11-04 01:13:18.882975994 +0000 UTC m=+967.641638154" Nov 04 01:13:19.414523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:19.414491 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:27.415083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:27.415017 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:27.427454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:27.427422 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:27.842564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:13:27.842528 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:13:43.832575 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:13:43.832537 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:14:07.495086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.495049 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm"] Nov 04 01:14:07.498948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.498927 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.556846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.556819 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:14:07.607532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.607503 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm"] Nov 04 01:14:07.639476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.639447 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.639648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.639627 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.639696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.639657 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5bf\" (UniqueName: \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.654803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.654778 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:07.740371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.740349 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.740506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.740375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5bf\" (UniqueName: \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.740506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.740410 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.741193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.741175 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.743138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.743112 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.794908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.794855 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5bf\" (UniqueName: \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\") pod \"maas-api-revoke-sub-38779493-kdrjm\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:07.813307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:07.813287 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:08.184479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:08.184431 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm"] Nov 04 01:14:08.185153 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:08.185114 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbcf9e2_ec91_4b3b_9478_c411f217c56b.slice/crio-274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6 WatchSource:0}: Error finding container 274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6: Status 404 returned error can't find the container with id 274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6 Nov 04 01:14:08.187546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:08.187526 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:14:09.148730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:09.148689 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" event={"ID":"1dbcf9e2-ec91-4b3b-9478-c411f217c56b","Type":"ContainerStarted","Data":"274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6"} Nov 04 01:14:13.186848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:13.186817 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" event={"ID":"1dbcf9e2-ec91-4b3b-9478-c411f217c56b","Type":"ContainerStarted","Data":"e41af80bff13daf68ca7d0305207d1f23de6f9312f32e388da394c5cee507fe6"} Nov 04 01:14:13.296949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:13.296904 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" podStartSLOduration=1.614511302 podStartE2EDuration="6.296890562s" totalImagesPullingTime="4.68237926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:13.294022657 +0000 UTC m=+1022.052684824" watchObservedRunningTime="2026-11-04 01:14:13.296890562 +0000 UTC m=+1022.055552723" Nov 04 01:14:13.834028 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:14:13.833971 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:14:14.196000 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:14.195954 4486 generic.go:356] "Generic (PLEG): container finished" podID="1dbcf9e2-ec91-4b3b-9478-c411f217c56b" containerID="e41af80bff13daf68ca7d0305207d1f23de6f9312f32e388da394c5cee507fe6" exitCode=0 Nov 04 01:14:14.196404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:14.196044 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" event={"ID":"1dbcf9e2-ec91-4b3b-9478-c411f217c56b","Type":"ContainerDied","Data":"e41af80bff13daf68ca7d0305207d1f23de6f9312f32e388da394c5cee507fe6"} Nov 04 01:14:15.344703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.344674 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:15.441636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.441612 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\" (UniqueName: \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\") pod \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " Nov 04 01:14:15.441774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.441668 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\") pod \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " Nov 04 01:14:15.441774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.441748 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\") pod \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\" (UID: \"1dbcf9e2-ec91-4b3b-9478-c411f217c56b\") " Nov 04 01:14:15.442089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.442064 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca" pod "1dbcf9e2-ec91-4b3b-9478-c411f217c56b" (UID: "1dbcf9e2-ec91-4b3b-9478-c411f217c56b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:15.443666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.443642 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf" pod "1dbcf9e2-ec91-4b3b-9478-c411f217c56b" (UID: "1dbcf9e2-ec91-4b3b-9478-c411f217c56b"). InnerVolumeSpecName "kube-api-access-nf5bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:15.443736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.443719 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca" pod "1dbcf9e2-ec91-4b3b-9478-c411f217c56b" (UID: "1dbcf9e2-ec91-4b3b-9478-c411f217c56b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:14:15.543077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.543020 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5bf\" (UniqueName: \"kubernetes.io/projected/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-kube-api-access-nf5bf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:15.543077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.543043 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:15.543077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:15.543054 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dbcf9e2-ec91-4b3b-9478-c411f217c56b-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:16.121266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.121232 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:14:16.121765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.121751 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1dbcf9e2-ec91-4b3b-9478-c411f217c56b" containerName="revoke-keys" Nov 04 01:14:16.121812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.121768 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbcf9e2-ec91-4b3b-9478-c411f217c56b" containerName="revoke-keys" Nov 04 01:14:16.121875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.121865 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbcf9e2-ec91-4b3b-9478-c411f217c56b" containerName="revoke-keys" Nov 04 01:14:16.125438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.125419 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148210 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148250 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148273 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148381 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148420 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148440 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148459 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljxl\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148497 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.148662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.148648 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.175258 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.175233 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e993d7-openshift-default-dockercfg-fr6p4\"" Nov 04 01:14:16.211366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.211347 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" Nov 04 01:14:16.211461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.211341 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm" event={"ID":"1dbcf9e2-ec91-4b3b-9478-c411f217c56b","Type":"ContainerDied","Data":"274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6"} Nov 04 01:14:16.211522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.211485 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="274356eb43c91f6628b8ee923232d80a90f2433106ffc9d90ec3573d428d38d6" Nov 04 01:14:16.226090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.226069 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:14:16.249232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249205 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249263 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249299 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249330 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249387 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249417 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249445 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljxl\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249470 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249533 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.249716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.249694 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.250245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.250214 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.250351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.250251 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.250351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.250304 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.250351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.250336 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.251559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.251540 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.252216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.252200 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.304271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.304249 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.304392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.304374 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljxl\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.439717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.439656 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:16.858566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.858531 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:14:16.859453 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:16.859418 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f016a1_5ed0_4a05_93be_fdacdaf00a3d.slice/crio-6caf6e654c5402a92d72b4354da4e1f162fe31c004d9b871471828e2191c2418 WatchSource:0}: Error finding container 6caf6e654c5402a92d72b4354da4e1f162fe31c004d9b871471828e2191c2418: Status 404 returned error can't find the container with id 6caf6e654c5402a92d72b4354da4e1f162fe31c004d9b871471828e2191c2418 Nov 04 01:14:16.862359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.862318 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:14:16.862474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.862401 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:14:16.862474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:16.862441 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:14:17.228676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.228575 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" event={"ID":"04f016a1-5ed0-4a05-93be-fdacdaf00a3d","Type":"ContainerStarted","Data":"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342"} Nov 04 01:14:17.228676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.228673 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" event={"ID":"04f016a1-5ed0-4a05-93be-fdacdaf00a3d","Type":"ContainerStarted","Data":"6caf6e654c5402a92d72b4354da4e1f162fe31c004d9b871471828e2191c2418"} Nov 04 01:14:17.352507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.352445 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podStartSLOduration=1.352426224 podStartE2EDuration="1.352426224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:17.351886943 +0000 UTC m=+1026.110549103" watchObservedRunningTime="2026-11-04 01:14:17.352426224 +0000 UTC m=+1026.111088385" Nov 04 01:14:17.452008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.447235 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:17.452008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.451078 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:17.452008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:17.451145 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:18.450008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:18.449353 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:18.450008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:18.449425 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:19.445008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:19.443302 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:19.445008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:19.443377 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:20.444006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:20.443314 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:20.444006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:20.443373 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:21.442373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:21.442336 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:21.443003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:21.442944 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:22.441353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:22.441315 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 04 01:14:22.441666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:22.441635 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 04 01:14:23.122401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.122308 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst"] Nov 04 01:14:23.126899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.126878 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.178090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.178057 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst"] Nov 04 01:14:23.221850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.221817 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:23.221850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.221845 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:14:23.229334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.229300 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.229462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.229352 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrjq\" (UniqueName: \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.229512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.229481 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.330896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.330850 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.331117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.330918 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrjq\" (UniqueName: \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.331117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.331039 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.331826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.331793 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.333639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.333615 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.385517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.385455 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrjq\" (UniqueName: \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\") pod \"maas-api-revoke-sub-2bbd59ee-4wdst\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.441719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.441686 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:23.558568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.558346 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:23.559467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.559286 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:23.560402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.560378 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:14:23.826501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:23.826473 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst"] Nov 04 01:14:23.828762 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:23.828731 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacd9545_7571_487f_8c54_cbe0babd07ae.slice/crio-8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7 WatchSource:0}: Error finding container 8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7: Status 404 returned error can't find the container with id 8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7 Nov 04 01:14:24.303853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:24.303809 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" event={"ID":"bacd9545-7571-487f-8c54-cbe0babd07ae","Type":"ContainerStarted","Data":"fe680930a3cfef9b8eae1db0da8299f5f414626abcc1cfb8dd82e5804a3a0d54"} Nov 04 01:14:24.304338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:24.303880 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" event={"ID":"bacd9545-7571-487f-8c54-cbe0babd07ae","Type":"ContainerStarted","Data":"8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7"} Nov 04 01:14:24.413973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:24.413880 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" podStartSLOduration=1.413866598 podStartE2EDuration="1.413866598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:24.410935221 +0000 UTC m=+1033.169597383" watchObservedRunningTime="2026-11-04 01:14:24.413866598 +0000 UTC m=+1033.172528758" Nov 04 01:14:25.312081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:25.312042 4486 generic.go:356] "Generic (PLEG): container finished" podID="bacd9545-7571-487f-8c54-cbe0babd07ae" containerID="fe680930a3cfef9b8eae1db0da8299f5f414626abcc1cfb8dd82e5804a3a0d54" exitCode=0 Nov 04 01:14:25.312556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:25.312084 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" event={"ID":"bacd9545-7571-487f-8c54-cbe0babd07ae","Type":"ContainerDied","Data":"fe680930a3cfef9b8eae1db0da8299f5f414626abcc1cfb8dd82e5804a3a0d54"} Nov 04 01:14:26.470865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.470837 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:26.572388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.572309 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\") pod \"bacd9545-7571-487f-8c54-cbe0babd07ae\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " Nov 04 01:14:26.572548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.572393 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\" (UniqueName: \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\") pod \"bacd9545-7571-487f-8c54-cbe0babd07ae\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " Nov 04 01:14:26.572548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.572435 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\") pod \"bacd9545-7571-487f-8c54-cbe0babd07ae\" (UID: \"bacd9545-7571-487f-8c54-cbe0babd07ae\") " Nov 04 01:14:26.572698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.572672 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca" pod "bacd9545-7571-487f-8c54-cbe0babd07ae" (UID: "bacd9545-7571-487f-8c54-cbe0babd07ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:26.572912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.572887 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:26.574506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.574482 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca" pod "bacd9545-7571-487f-8c54-cbe0babd07ae" (UID: "bacd9545-7571-487f-8c54-cbe0babd07ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:14:26.574626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.574584 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq" pod "bacd9545-7571-487f-8c54-cbe0babd07ae" (UID: "bacd9545-7571-487f-8c54-cbe0babd07ae"). InnerVolumeSpecName "kube-api-access-qmrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:26.673728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.673696 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrjq\" (UniqueName: \"kubernetes.io/projected/bacd9545-7571-487f-8c54-cbe0babd07ae-kube-api-access-qmrjq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:26.673728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:26.673724 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bacd9545-7571-487f-8c54-cbe0babd07ae-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:27.326551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:27.326516 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" Nov 04 01:14:27.326551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:27.326527 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst" event={"ID":"bacd9545-7571-487f-8c54-cbe0babd07ae","Type":"ContainerDied","Data":"8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7"} Nov 04 01:14:27.326760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:27.326584 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dca18ad2b1fad9e8598a6289e8e810aac6514d60ded1143e0f463844ed395f7" Nov 04 01:14:33.091480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.091443 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4"] Nov 04 01:14:33.092289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.092265 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bacd9545-7571-487f-8c54-cbe0babd07ae" containerName="revoke-keys" Nov 04 01:14:33.092289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.092289 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bacd9545-7571-487f-8c54-cbe0babd07ae" containerName="revoke-keys" Nov 04 01:14:33.092469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.092442 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bacd9545-7571-487f-8c54-cbe0babd07ae" containerName="revoke-keys" Nov 04 01:14:33.096049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.096028 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.149808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.149777 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4"] Nov 04 01:14:33.192464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.192432 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:33.192623 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.192447 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:14:33.249457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.249426 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.249616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.249479 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.249662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.249618 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bj8p\" (UniqueName: \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.350871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.350788 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.350871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.350863 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.351146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.350957 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bj8p\" (UniqueName: \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.351660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.351638 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.353277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.353249 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.405640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.405604 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bj8p\" (UniqueName: \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\") pod \"maas-api-revoke-sub-f3a2927b-c9vc4\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.408204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.408185 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:33.786569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:33.786537 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4"] Nov 04 01:14:33.786871 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:33.786848 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ee568c_3982_4bea_9628_85b39af42195.slice/crio-47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32 WatchSource:0}: Error finding container 47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32: Status 404 returned error can't find the container with id 47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32 Nov 04 01:14:34.372522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:34.372489 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" event={"ID":"73ee568c-3982-4bea-9628-85b39af42195","Type":"ContainerStarted","Data":"7a076ac8a455ccfd328009f98365222914b10e346b792521b0ef177e615e267e"} Nov 04 01:14:34.372884 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:34.372541 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" event={"ID":"73ee568c-3982-4bea-9628-85b39af42195","Type":"ContainerStarted","Data":"47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32"} Nov 04 01:14:34.481094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:34.481033 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" podStartSLOduration=1.481015449 podStartE2EDuration="1.481015449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:34.480247452 +0000 UTC m=+1043.238909617" watchObservedRunningTime="2026-11-04 01:14:34.481015449 +0000 UTC m=+1043.239677611" Nov 04 01:14:35.379667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:35.379632 4486 generic.go:356] "Generic (PLEG): container finished" podID="73ee568c-3982-4bea-9628-85b39af42195" containerID="7a076ac8a455ccfd328009f98365222914b10e346b792521b0ef177e615e267e" exitCode=0 Nov 04 01:14:35.380107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:35.379719 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" event={"ID":"73ee568c-3982-4bea-9628-85b39af42195","Type":"ContainerDied","Data":"7a076ac8a455ccfd328009f98365222914b10e346b792521b0ef177e615e267e"} Nov 04 01:14:36.532640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.532612 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:36.689512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.689425 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\") pod \"73ee568c-3982-4bea-9628-85b39af42195\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " Nov 04 01:14:36.689512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.689497 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\") pod \"73ee568c-3982-4bea-9628-85b39af42195\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " Nov 04 01:14:36.689700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.689537 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\" (UniqueName: \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\") pod \"73ee568c-3982-4bea-9628-85b39af42195\" (UID: \"73ee568c-3982-4bea-9628-85b39af42195\") " Nov 04 01:14:36.689919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.689891 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca" pod "73ee568c-3982-4bea-9628-85b39af42195" (UID: "73ee568c-3982-4bea-9628-85b39af42195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:36.691498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.691468 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca" pod "73ee568c-3982-4bea-9628-85b39af42195" (UID: "73ee568c-3982-4bea-9628-85b39af42195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:14:36.691669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.691649 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p" pod "73ee568c-3982-4bea-9628-85b39af42195" (UID: "73ee568c-3982-4bea-9628-85b39af42195"). InnerVolumeSpecName "kube-api-access-4bj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:36.791171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.791144 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ee568c-3982-4bea-9628-85b39af42195-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:36.791171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.791169 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ee568c-3982-4bea-9628-85b39af42195-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:36.791313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:36.791180 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bj8p\" (UniqueName: \"kubernetes.io/projected/73ee568c-3982-4bea-9628-85b39af42195-kube-api-access-4bj8p\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:37.393491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:37.393451 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" event={"ID":"73ee568c-3982-4bea-9628-85b39af42195","Type":"ContainerDied","Data":"47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32"} Nov 04 01:14:37.393624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:37.393504 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f04e9fa9c75cb2906dcc89e97dc237905b1c0568941cdd4fdb97b5ba3a0b32" Nov 04 01:14:37.393624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:37.393462 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4" Nov 04 01:14:38.136489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.136451 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:14:38.137332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.137311 4486 cpu_manager.go:414] "containerMap: removing container" podUID="73ee568c-3982-4bea-9628-85b39af42195" containerName="revoke-keys" Nov 04 01:14:38.137393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.137335 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ee568c-3982-4bea-9628-85b39af42195" containerName="revoke-keys" Nov 04 01:14:38.137495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.137484 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ee568c-3982-4bea-9628-85b39af42195" containerName="revoke-keys" Nov 04 01:14:38.140213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.140193 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.193367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.193332 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:14:38.234606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.234572 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"premium-w3-2ee045-kserve-self-signed-certs\"" Nov 04 01:14:38.234763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.234645 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"kube-root-ca.crt\"" Nov 04 01:14:38.234763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.234643 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"default-dockercfg-pjk8b\"" Nov 04 01:14:38.234847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.234761 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2ee045\"/\"openshift-service-ca.crt\"" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.306967 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qk5\" (UniqueName: \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.307165 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.307216 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.307251 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.307343 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.311026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.307444 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.408767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408688 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.408767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408742 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qk5\" (UniqueName: \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408797 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408821 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408841 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.408902 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.409191 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.409250 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.409322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.409292 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.410969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.410948 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.411291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.411275 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.464416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.464383 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qk5\" (UniqueName: \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\") pod \"premium-w3-2ee045-kserve-5885fb4578-ldkh2\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.753905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.753814 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:38.842761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.842733 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn"] Nov 04 01:14:38.850278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.850255 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:38.884303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.884276 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn"] Nov 04 01:14:38.900604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.900555 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:38.900604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:38.900571 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:14:39.015343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.015270 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.015494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.015410 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.015560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.015515 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrh8\" (UniqueName: \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.117108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.117075 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.117241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.117147 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.117241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.117197 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrh8\" (UniqueName: \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.117792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.117771 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.119362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.119341 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.127277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.127218 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:14:39.127722 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:39.127700 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod829b4a50_3c8a_4531_924a_837c0b9bac8b.slice/crio-e49817c8175c6543e358f76569c5481cb4bf9e9137ec73d0c581d40adfe3acc2 WatchSource:0}: Error finding container e49817c8175c6543e358f76569c5481cb4bf9e9137ec73d0c581d40adfe3acc2: Status 404 returned error can't find the container with id e49817c8175c6543e358f76569c5481cb4bf9e9137ec73d0c581d40adfe3acc2 Nov 04 01:14:39.171648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.171624 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrh8\" (UniqueName: \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\") pod \"maas-api-revoke-sub-c6b4fa56-9mfhn\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.407212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.407171 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" event={"ID":"829b4a50-3c8a-4531-924a-837c0b9bac8b","Type":"ContainerStarted","Data":"4975df01c498c89ba5796ef032b37b94ff01a08eb0f626e1da5b668ec742f89c"} Nov 04 01:14:39.407358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.407222 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" event={"ID":"829b4a50-3c8a-4531-924a-837c0b9bac8b","Type":"ContainerStarted","Data":"e49817c8175c6543e358f76569c5481cb4bf9e9137ec73d0c581d40adfe3acc2"} Nov 04 01:14:39.470744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.470715 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:39.516612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.516544 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podStartSLOduration=1.5165249809999999 podStartE2EDuration="1.516524981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:39.515845354 +0000 UTC m=+1048.274507517" watchObservedRunningTime="2026-11-04 01:14:39.516524981 +0000 UTC m=+1048.275187145" Nov 04 01:14:39.754587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.754507 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:39.848498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:39.848469 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn"] Nov 04 01:14:39.849193 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:39.849146 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b75b13_0217_45cf_97b0_32d0ccec919e.slice/crio-2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5 WatchSource:0}: Error finding container 2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5: Status 404 returned error can't find the container with id 2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5 Nov 04 01:14:40.416574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:40.416530 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" event={"ID":"59b75b13-0217-45cf-97b0-32d0ccec919e","Type":"ContainerStarted","Data":"17be05eaf29f8f1f7722e90a834228d436af7fbffe8e927e379d14773b679082"} Nov 04 01:14:40.416967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:40.416583 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" event={"ID":"59b75b13-0217-45cf-97b0-32d0ccec919e","Type":"ContainerStarted","Data":"2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5"} Nov 04 01:14:40.527082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:40.527028 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" podStartSLOduration=2.527013597 podStartE2EDuration="2.527013597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:40.525211471 +0000 UTC m=+1049.283873632" watchObservedRunningTime="2026-11-04 01:14:40.527013597 +0000 UTC m=+1049.285675757" Nov 04 01:14:41.429272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:41.428059 4486 generic.go:356] "Generic (PLEG): container finished" podID="59b75b13-0217-45cf-97b0-32d0ccec919e" containerID="17be05eaf29f8f1f7722e90a834228d436af7fbffe8e927e379d14773b679082" exitCode=0 Nov 04 01:14:41.429272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:41.428125 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" event={"ID":"59b75b13-0217-45cf-97b0-32d0ccec919e","Type":"ContainerDied","Data":"17be05eaf29f8f1f7722e90a834228d436af7fbffe8e927e379d14773b679082"} Nov 04 01:14:42.612681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.612656 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:42.770358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.770279 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\" (UniqueName: \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\") pod \"59b75b13-0217-45cf-97b0-32d0ccec919e\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " Nov 04 01:14:42.770358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.770322 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\") pod \"59b75b13-0217-45cf-97b0-32d0ccec919e\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " Nov 04 01:14:42.770591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.770462 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\") pod \"59b75b13-0217-45cf-97b0-32d0ccec919e\" (UID: \"59b75b13-0217-45cf-97b0-32d0ccec919e\") " Nov 04 01:14:42.770913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.770874 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca" pod "59b75b13-0217-45cf-97b0-32d0ccec919e" (UID: "59b75b13-0217-45cf-97b0-32d0ccec919e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:42.772572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.772542 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8" pod "59b75b13-0217-45cf-97b0-32d0ccec919e" (UID: "59b75b13-0217-45cf-97b0-32d0ccec919e"). InnerVolumeSpecName "kube-api-access-hdrh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:42.772724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.772709 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca" pod "59b75b13-0217-45cf-97b0-32d0ccec919e" (UID: "59b75b13-0217-45cf-97b0-32d0ccec919e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:14:42.871828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.871787 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrh8\" (UniqueName: \"kubernetes.io/projected/59b75b13-0217-45cf-97b0-32d0ccec919e-kube-api-access-hdrh8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:42.871828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.871818 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:42.871828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:42.871828 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59b75b13-0217-45cf-97b0-32d0ccec919e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:43.445055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.445025 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" Nov 04 01:14:43.445055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.445035 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn" event={"ID":"59b75b13-0217-45cf-97b0-32d0ccec919e","Type":"ContainerDied","Data":"2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5"} Nov 04 01:14:43.445263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.445101 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce73083fa0a1a764df0e01ef2912f60ef60dfbc297e7981975eb0da935bb5b5" Nov 04 01:14:43.835326 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:14:43.835292 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:14:43.909789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.909746 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q"] Nov 04 01:14:43.910712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.910690 4486 cpu_manager.go:414] "containerMap: removing container" podUID="59b75b13-0217-45cf-97b0-32d0ccec919e" containerName="revoke-keys" Nov 04 01:14:43.910788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.910714 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b75b13-0217-45cf-97b0-32d0ccec919e" containerName="revoke-keys" Nov 04 01:14:43.910916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.910904 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b75b13-0217-45cf-97b0-32d0ccec919e" containerName="revoke-keys" Nov 04 01:14:43.913765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.913736 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:43.964524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.964493 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q"] Nov 04 01:14:43.985807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.985770 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:43.986008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.985951 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:43.986096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:43.986007 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wblt\" (UniqueName: \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.008204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.008171 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:14:44.008338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.008238 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:14:44.087033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.086939 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.087171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.087078 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.087171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.087104 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wblt\" (UniqueName: \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.087734 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.087713 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.089416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.089396 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.143776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.143742 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wblt\" (UniqueName: \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\") pod \"maas-api-revoke-sub-21cf5835-p5m6q\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.226485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.226453 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:44.599975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:44.599937 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q"] Nov 04 01:14:44.601059 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:14:44.600722 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97644f5_d5d5_435b_a2ed_e493e82e99c7.slice/crio-000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4 WatchSource:0}: Error finding container 000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4: Status 404 returned error can't find the container with id 000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4 Nov 04 01:14:45.463407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:45.463361 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" event={"ID":"f97644f5-d5d5-435b-a2ed-e493e82e99c7","Type":"ContainerStarted","Data":"fb448a1c05413e44a91220d0738ad47dc1b55cf2665ff696aebf556b6c6e2c0f"} Nov 04 01:14:45.463804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:45.463423 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" event={"ID":"f97644f5-d5d5-435b-a2ed-e493e82e99c7","Type":"ContainerStarted","Data":"000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4"} Nov 04 01:14:45.573416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:45.573368 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" podStartSLOduration=2.5733549079999998 podStartE2EDuration="2.573354908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:45.570992934 +0000 UTC m=+1054.329655089" watchObservedRunningTime="2026-11-04 01:14:45.573354908 +0000 UTC m=+1054.332017069" Nov 04 01:14:46.471505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:46.471467 4486 generic.go:356] "Generic (PLEG): container finished" podID="f97644f5-d5d5-435b-a2ed-e493e82e99c7" containerID="fb448a1c05413e44a91220d0738ad47dc1b55cf2665ff696aebf556b6c6e2c0f" exitCode=0 Nov 04 01:14:46.471931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:46.471556 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" event={"ID":"f97644f5-d5d5-435b-a2ed-e493e82e99c7","Type":"ContainerDied","Data":"fb448a1c05413e44a91220d0738ad47dc1b55cf2665ff696aebf556b6c6e2c0f"} Nov 04 01:14:47.628880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.628855 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:47.725993 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.725948 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\") pod \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " Nov 04 01:14:47.726171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.726067 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\") pod \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " Nov 04 01:14:47.726171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.726110 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\" (UniqueName: \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\") pod \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\" (UID: \"f97644f5-d5d5-435b-a2ed-e493e82e99c7\") " Nov 04 01:14:47.726408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.726383 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca" pod "f97644f5-d5d5-435b-a2ed-e493e82e99c7" (UID: "f97644f5-d5d5-435b-a2ed-e493e82e99c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:14:47.728135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.728070 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca" pod "f97644f5-d5d5-435b-a2ed-e493e82e99c7" (UID: "f97644f5-d5d5-435b-a2ed-e493e82e99c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:14:47.728228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.728155 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt" pod "f97644f5-d5d5-435b-a2ed-e493e82e99c7" (UID: "f97644f5-d5d5-435b-a2ed-e493e82e99c7"). InnerVolumeSpecName "kube-api-access-4wblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:14:47.826927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.826898 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wblt\" (UniqueName: \"kubernetes.io/projected/f97644f5-d5d5-435b-a2ed-e493e82e99c7-kube-api-access-4wblt\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:47.826927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.826923 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:47.827105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:47.826933 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97644f5-d5d5-435b-a2ed-e493e82e99c7-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:14:48.485243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:48.485209 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" Nov 04 01:14:48.485408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:48.485289 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q" event={"ID":"f97644f5-d5d5-435b-a2ed-e493e82e99c7","Type":"ContainerDied","Data":"000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4"} Nov 04 01:14:48.485408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:48.485362 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="000d2036fb25b4ca4a72c39e440e2be6fe3c610a04f188d0d4e2222e8e7e76c4" Nov 04 01:14:48.754766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:48.754694 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:48.767714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:48.767682 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:14:49.502627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:14:49.502599 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:15:00.169638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.169588 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:00.170273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.170252 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f97644f5-d5d5-435b-a2ed-e493e82e99c7" containerName="revoke-keys" Nov 04 01:15:00.170273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.170273 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97644f5-d5d5-435b-a2ed-e493e82e99c7" containerName="revoke-keys" Nov 04 01:15:00.170435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.170373 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97644f5-d5d5-435b-a2ed-e493e82e99c7" containerName="revoke-keys" Nov 04 01:15:00.174336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.174316 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.175058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.175032 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:00.179728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.179706 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.179933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.179903 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:00.184868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.184731 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.221874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.221849 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:00.224197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.224164 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:15:00.226825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.226801 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:00.226936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.226917 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.231296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.231273 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:00.231411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.231389 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.235656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.235639 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.267341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.267315 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:00.273104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.272754 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:00.280632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.280606 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:00.284196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.284172 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:00.304714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304686 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.304874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304724 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7g4\" (UniqueName: \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.304874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304745 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcg6n\" (UniqueName: \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.304874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304844 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jsl\" (UniqueName: \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.304874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304872 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304889 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkk5\" (UniqueName: \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304914 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.304945 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pj4l\" (UniqueName: \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.305024 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.305063 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsfm\" (UniqueName: \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.305115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.305096 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.305420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.305129 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.314910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.314817 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:00.319598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.319575 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:00.406382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406344 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.406559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406434 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pj4l\" (UniqueName: \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.406559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406522 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.406770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406585 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsfm\" (UniqueName: \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.406770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406637 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.406770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406713 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.406949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406797 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.406949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406859 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7g4\" (UniqueName: \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.406949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406885 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcg6n\" (UniqueName: \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.406949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406942 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jsl\" (UniqueName: \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.407186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.406970 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.407186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.407029 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkk5\" (UniqueName: \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.407297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.407247 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.407374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.407341 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.407866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.407653 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.408005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.407924 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.408258 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.408235 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.408309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.408265 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.462353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.461609 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pj4l\" (UniqueName: \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\") pod \"maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.462353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.461883 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jsl\" (UniqueName: \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\") pod \"maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.462353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.462260 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsfm\" (UniqueName: \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\") pod \"maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.462600 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.462492 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcg6n\" (UniqueName: \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.463082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.463054 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7g4\" (UniqueName: \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.464104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.464059 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkk5\" (UniqueName: \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\") pod \"maas-api-key-cleanup-29895915-zkkdn\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.490779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.490753 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:00.500244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.500045 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:00.525167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.525141 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:00.554626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.551692 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:00.554626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.552064 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:00.561226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.557407 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:00.921095 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:00.921066 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba68bb44_72c8_4dc4_8040_3ffd18d9f019.slice/crio-93159679eab0d810dcef2a5daf22471b87f59c479c6ca431a53acae48d15119c WatchSource:0}: Error finding container 93159679eab0d810dcef2a5daf22471b87f59c479c6ca431a53acae48d15119c: Status 404 returned error can't find the container with id 93159679eab0d810dcef2a5daf22471b87f59c479c6ca431a53acae48d15119c Nov 04 01:15:00.922777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.922748 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:00.968255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.968223 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:00.968956 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:00.968929 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77de7f8b_1f5b_4edc_b7f1_e06914f72c1d.slice/crio-cdd1e12cfb06743684584680a265eedda295036bea9a2db72e743194aa86d4c9 WatchSource:0}: Error finding container cdd1e12cfb06743684584680a265eedda295036bea9a2db72e743194aa86d4c9: Status 404 returned error can't find the container with id cdd1e12cfb06743684584680a265eedda295036bea9a2db72e743194aa86d4c9 Nov 04 01:15:00.971805 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:00.971774 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0150afb8_84e5_4cd1_a8ae_e3fc691518e9.slice/crio-0600fd0fa125dd242c916e28ec157c5eeef4f52a7ace8f799641b107d33d0ff2 WatchSource:0}: Error finding container 0600fd0fa125dd242c916e28ec157c5eeef4f52a7ace8f799641b107d33d0ff2: Status 404 returned error can't find the container with id 0600fd0fa125dd242c916e28ec157c5eeef4f52a7ace8f799641b107d33d0ff2 Nov 04 01:15:00.972361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:00.972342 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:01.017562 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:01.017533 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d94d58_ae32_4257_b0de_d545c4ab9f40.slice/crio-e34403b155c8d0251621a14fef47ac92db68de1c296a1944373b407ef2fd859b WatchSource:0}: Error finding container e34403b155c8d0251621a14fef47ac92db68de1c296a1944373b407ef2fd859b: Status 404 returned error can't find the container with id e34403b155c8d0251621a14fef47ac92db68de1c296a1944373b407ef2fd859b Nov 04 01:15:01.018424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.018405 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:01.021008 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:01.020938 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c7b28d_82a9_4129_a52e_7cff71dc90f0.slice/crio-ddacee379af3bc351d5855a08ea505f17cef58d7e3af42ffcaed23c04aa915c9 WatchSource:0}: Error finding container ddacee379af3bc351d5855a08ea505f17cef58d7e3af42ffcaed23c04aa915c9: Status 404 returned error can't find the container with id ddacee379af3bc351d5855a08ea505f17cef58d7e3af42ffcaed23c04aa915c9 Nov 04 01:15:01.023083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.023060 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:01.030543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.030234 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:01.132730 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:01.132697 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da3cf5f_208f_44d5_b2bb_5e4c7ddda982.slice/crio-db18a935a60a14c0d8348ea1b159b9c64b8a291829e0695577810bc609fcfc75 WatchSource:0}: Error finding container db18a935a60a14c0d8348ea1b159b9c64b8a291829e0695577810bc609fcfc75: Status 404 returned error can't find the container with id db18a935a60a14c0d8348ea1b159b9c64b8a291829e0695577810bc609fcfc75 Nov 04 01:15:01.651705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.651642 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerStarted","Data":"ddacee379af3bc351d5855a08ea505f17cef58d7e3af42ffcaed23c04aa915c9"} Nov 04 01:15:01.653929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.653711 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerStarted","Data":"1e42fb563852c206cc999a76f14597129e7cce3694e079eb4ea0228e541f8f4e"} Nov 04 01:15:01.653929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.653762 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerStarted","Data":"0600fd0fa125dd242c916e28ec157c5eeef4f52a7ace8f799641b107d33d0ff2"} Nov 04 01:15:01.655822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.655788 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerStarted","Data":"5c2501bfcb1a24f2edb26e8ff68faea330a385e8bf1746101362b7ae0cce78bb"} Nov 04 01:15:01.655921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.655838 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerStarted","Data":"db18a935a60a14c0d8348ea1b159b9c64b8a291829e0695577810bc609fcfc75"} Nov 04 01:15:01.657472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.657449 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerStarted","Data":"bd1e4744d60281e9b9fa4cd6b0d1e7f8a37cf2480f90ae65e246ceadf5701c92"} Nov 04 01:15:01.657566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.657494 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerStarted","Data":"cdd1e12cfb06743684584680a265eedda295036bea9a2db72e743194aa86d4c9"} Nov 04 01:15:01.659077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.659049 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerStarted","Data":"2e76911f84d392270596385bc6a360f693b0fe76276fd90a4efe23f5464cd6db"} Nov 04 01:15:01.659180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.659092 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerStarted","Data":"93159679eab0d810dcef2a5daf22471b87f59c479c6ca431a53acae48d15119c"} Nov 04 01:15:01.666118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.664639 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerStarted","Data":"e1c812505864e91db6b1a1421b2eaefe76b3fa217949a90b07f223ba446954db"} Nov 04 01:15:01.666118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.664690 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerStarted","Data":"e34403b155c8d0251621a14fef47ac92db68de1c296a1944373b407ef2fd859b"} Nov 04 01:15:01.761331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.761273 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" podStartSLOduration=1.761260483 podStartE2EDuration="1.761260483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:01.759120827 +0000 UTC m=+1070.517782987" watchObservedRunningTime="2026-11-04 01:15:01.761260483 +0000 UTC m=+1070.519922644" Nov 04 01:15:01.869263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.869205 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" podStartSLOduration=1.869191743 podStartE2EDuration="1.869191743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:01.866659117 +0000 UTC m=+1070.625321278" watchObservedRunningTime="2026-11-04 01:15:01.869191743 +0000 UTC m=+1070.627853905" Nov 04 01:15:01.975701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:01.975598 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" podStartSLOduration=1.9755856939999998 podStartE2EDuration="1.975585694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:01.973857841 +0000 UTC m=+1070.732520003" watchObservedRunningTime="2026-11-04 01:15:01.975585694 +0000 UTC m=+1070.734247895" Nov 04 01:15:02.083312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.083260 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" podStartSLOduration=2.083247015 podStartE2EDuration="2.083247015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:02.081778239 +0000 UTC m=+1070.840440401" watchObservedRunningTime="2026-11-04 01:15:02.083247015 +0000 UTC m=+1070.841909175" Nov 04 01:15:02.303112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.303065 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" podStartSLOduration=2.303050482 podStartE2EDuration="2.303050482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:02.30223067 +0000 UTC m=+1071.060892832" watchObservedRunningTime="2026-11-04 01:15:02.303050482 +0000 UTC m=+1071.061712642" Nov 04 01:15:02.305224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.305191 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" podStartSLOduration=2.305178904 podStartE2EDuration="2.305178904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:02.195136146 +0000 UTC m=+1070.953798310" watchObservedRunningTime="2026-11-04 01:15:02.305178904 +0000 UTC m=+1071.063841163" Nov 04 01:15:02.678220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.678188 4486 generic.go:356] "Generic (PLEG): container finished" podID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerID="d360cb46cc550e51189acd08eb679ed0d8f255844a61eacdda6e834c9639a0cb" exitCode=60 Nov 04 01:15:02.678705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.678266 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerDied","Data":"d360cb46cc550e51189acd08eb679ed0d8f255844a61eacdda6e834c9639a0cb"} Nov 04 01:15:02.678705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.678517 4486 scope.go:122] "RemoveContainer" containerID="d360cb46cc550e51189acd08eb679ed0d8f255844a61eacdda6e834c9639a0cb" Nov 04 01:15:02.680166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.680147 4486 generic.go:356] "Generic (PLEG): container finished" podID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerID="1e42fb563852c206cc999a76f14597129e7cce3694e079eb4ea0228e541f8f4e" exitCode=60 Nov 04 01:15:02.680255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.680229 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerDied","Data":"1e42fb563852c206cc999a76f14597129e7cce3694e079eb4ea0228e541f8f4e"} Nov 04 01:15:02.680451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.680437 4486 scope.go:122] "RemoveContainer" containerID="1e42fb563852c206cc999a76f14597129e7cce3694e079eb4ea0228e541f8f4e" Nov 04 01:15:02.682045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.682025 4486 generic.go:356] "Generic (PLEG): container finished" podID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerID="5c2501bfcb1a24f2edb26e8ff68faea330a385e8bf1746101362b7ae0cce78bb" exitCode=60 Nov 04 01:15:02.682412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.682108 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerDied","Data":"5c2501bfcb1a24f2edb26e8ff68faea330a385e8bf1746101362b7ae0cce78bb"} Nov 04 01:15:02.682412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.682282 4486 scope.go:122] "RemoveContainer" containerID="5c2501bfcb1a24f2edb26e8ff68faea330a385e8bf1746101362b7ae0cce78bb" Nov 04 01:15:02.683940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.683916 4486 generic.go:356] "Generic (PLEG): container finished" podID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerID="bd1e4744d60281e9b9fa4cd6b0d1e7f8a37cf2480f90ae65e246ceadf5701c92" exitCode=60 Nov 04 01:15:02.684065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.684042 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerDied","Data":"bd1e4744d60281e9b9fa4cd6b0d1e7f8a37cf2480f90ae65e246ceadf5701c92"} Nov 04 01:15:02.684212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.684198 4486 scope.go:122] "RemoveContainer" containerID="bd1e4744d60281e9b9fa4cd6b0d1e7f8a37cf2480f90ae65e246ceadf5701c92" Nov 04 01:15:02.685565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.685547 4486 generic.go:356] "Generic (PLEG): container finished" podID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerID="2e76911f84d392270596385bc6a360f693b0fe76276fd90a4efe23f5464cd6db" exitCode=60 Nov 04 01:15:02.685661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.685625 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerDied","Data":"2e76911f84d392270596385bc6a360f693b0fe76276fd90a4efe23f5464cd6db"} Nov 04 01:15:02.685794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.685780 4486 scope.go:122] "RemoveContainer" containerID="2e76911f84d392270596385bc6a360f693b0fe76276fd90a4efe23f5464cd6db" Nov 04 01:15:02.687127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.687108 4486 generic.go:356] "Generic (PLEG): container finished" podID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerID="e1c812505864e91db6b1a1421b2eaefe76b3fa217949a90b07f223ba446954db" exitCode=60 Nov 04 01:15:02.687218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.687175 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerDied","Data":"e1c812505864e91db6b1a1421b2eaefe76b3fa217949a90b07f223ba446954db"} Nov 04 01:15:02.687318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:02.687304 4486 scope.go:122] "RemoveContainer" containerID="e1c812505864e91db6b1a1421b2eaefe76b3fa217949a90b07f223ba446954db" Nov 04 01:15:03.695465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.695423 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerStarted","Data":"d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec"} Nov 04 01:15:03.697950 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.697901 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerStarted","Data":"338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac"} Nov 04 01:15:03.699975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.699953 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerStarted","Data":"20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186"} Nov 04 01:15:03.701937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.701915 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerStarted","Data":"b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6"} Nov 04 01:15:03.703795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.703766 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerStarted","Data":"05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6"} Nov 04 01:15:03.705669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:03.705646 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerStarted","Data":"99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841"} Nov 04 01:15:04.714976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.714944 4486 generic.go:356] "Generic (PLEG): container finished" podID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerID="05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6" exitCode=60 Nov 04 01:15:04.715504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.715013 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerDied","Data":"05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6"} Nov 04 01:15:04.715504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.715084 4486 scope.go:122] "RemoveContainer" containerID="d360cb46cc550e51189acd08eb679ed0d8f255844a61eacdda6e834c9639a0cb" Nov 04 01:15:04.715504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.715365 4486 scope.go:122] "RemoveContainer" containerID="05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6" Nov 04 01:15:04.715681 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.715665 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895915-zkkdn_odh-ai-gateway-infra(58c7b28d-82a9-4129-a52e-7cff71dc90f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" Nov 04 01:15:04.717035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.717012 4486 generic.go:356] "Generic (PLEG): container finished" podID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerID="99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841" exitCode=60 Nov 04 01:15:04.717122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.717082 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerDied","Data":"99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841"} Nov 04 01:15:04.717313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.717299 4486 scope.go:122] "RemoveContainer" containerID="99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841" Nov 04 01:15:04.717610 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.717587 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728_odh-ai-gateway-infra(0150afb8-84e5-4cd1-a8ae-e3fc691518e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" Nov 04 01:15:04.718943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.718921 4486 generic.go:356] "Generic (PLEG): container finished" podID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerID="d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec" exitCode=60 Nov 04 01:15:04.719055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.719015 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerDied","Data":"d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec"} Nov 04 01:15:04.719316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.719299 4486 scope.go:122] "RemoveContainer" containerID="d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec" Nov 04 01:15:04.719550 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.719524 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz_odh-ai-gateway-infra(3da3cf5f-208f-44d5-b2bb-5e4c7ddda982)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" Nov 04 01:15:04.720793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.720773 4486 generic.go:356] "Generic (PLEG): container finished" podID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerID="338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac" exitCode=60 Nov 04 01:15:04.720889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.720853 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerDied","Data":"338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac"} Nov 04 01:15:04.721071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.721057 4486 scope.go:122] "RemoveContainer" containerID="338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac" Nov 04 01:15:04.721288 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.721267 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj_odh-ai-gateway-infra(77de7f8b-1f5b-4edc-b7f1-e06914f72c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" Nov 04 01:15:04.722762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.722744 4486 generic.go:356] "Generic (PLEG): container finished" podID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerID="20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186" exitCode=60 Nov 04 01:15:04.722853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.722813 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerDied","Data":"20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186"} Nov 04 01:15:04.723007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.722974 4486 scope.go:122] "RemoveContainer" containerID="20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186" Nov 04 01:15:04.723239 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.723218 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc_odh-ai-gateway-infra(ba68bb44-72c8-4dc4-8040-3ffd18d9f019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" Nov 04 01:15:04.724694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.724676 4486 generic.go:356] "Generic (PLEG): container finished" podID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerID="b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6" exitCode=60 Nov 04 01:15:04.724779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.724749 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerDied","Data":"b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6"} Nov 04 01:15:04.724894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.724882 4486 scope.go:122] "RemoveContainer" containerID="b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6" Nov 04 01:15:04.725084 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:04.725067 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g_odh-ai-gateway-infra(77d94d58-ae32-4257-b0de-d545c4ab9f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" Nov 04 01:15:04.728706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.728690 4486 scope.go:122] "RemoveContainer" containerID="1e42fb563852c206cc999a76f14597129e7cce3694e079eb4ea0228e541f8f4e" Nov 04 01:15:04.749393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.749369 4486 scope.go:122] "RemoveContainer" containerID="5c2501bfcb1a24f2edb26e8ff68faea330a385e8bf1746101362b7ae0cce78bb" Nov 04 01:15:04.764766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.764742 4486 scope.go:122] "RemoveContainer" containerID="bd1e4744d60281e9b9fa4cd6b0d1e7f8a37cf2480f90ae65e246ceadf5701c92" Nov 04 01:15:04.783131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.783088 4486 scope.go:122] "RemoveContainer" containerID="2e76911f84d392270596385bc6a360f693b0fe76276fd90a4efe23f5464cd6db" Nov 04 01:15:04.796943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:04.796921 4486 scope.go:122] "RemoveContainer" containerID="e1c812505864e91db6b1a1421b2eaefe76b3fa217949a90b07f223ba446954db" Nov 04 01:15:05.741647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.741618 4486 scope.go:122] "RemoveContainer" containerID="05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6" Nov 04 01:15:05.742142 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.741942 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895915-zkkdn_odh-ai-gateway-infra(58c7b28d-82a9-4129-a52e-7cff71dc90f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" Nov 04 01:15:05.743304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.743283 4486 scope.go:122] "RemoveContainer" containerID="99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841" Nov 04 01:15:05.743530 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.743505 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728_odh-ai-gateway-infra(0150afb8-84e5-4cd1-a8ae-e3fc691518e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" Nov 04 01:15:05.745113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.745093 4486 scope.go:122] "RemoveContainer" containerID="d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec" Nov 04 01:15:05.745321 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.745301 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz_odh-ai-gateway-infra(3da3cf5f-208f-44d5-b2bb-5e4c7ddda982)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" Nov 04 01:15:05.746774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.746753 4486 scope.go:122] "RemoveContainer" containerID="338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac" Nov 04 01:15:05.746955 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.746936 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj_odh-ai-gateway-infra(77de7f8b-1f5b-4edc-b7f1-e06914f72c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" Nov 04 01:15:05.748506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.748484 4486 scope.go:122] "RemoveContainer" containerID="20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186" Nov 04 01:15:05.748707 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.748686 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc_odh-ai-gateway-infra(ba68bb44-72c8-4dc4-8040-3ffd18d9f019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" Nov 04 01:15:05.750105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:05.750089 4486 scope.go:122] "RemoveContainer" containerID="b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6" Nov 04 01:15:05.750259 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:05.750241 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g_odh-ai-gateway-infra(77d94d58-ae32-4257-b0de-d545c4ab9f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" Nov 04 01:15:13.831251 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:13.831208 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:15:14.342062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:14.342030 4486 scope.go:122] "RemoveContainer" containerID="20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186" Nov 04 01:15:14.817782 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:14.817743 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerStarted","Data":"cafe741e683343cf390b6ac9db95403ee5f312e1684cad96266988179395c0b9"} Nov 04 01:15:15.347572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.347481 4486 scope.go:122] "RemoveContainer" containerID="d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec" Nov 04 01:15:15.827898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.827863 4486 generic.go:356] "Generic (PLEG): container finished" podID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerID="cafe741e683343cf390b6ac9db95403ee5f312e1684cad96266988179395c0b9" exitCode=60 Nov 04 01:15:15.828106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.827924 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerDied","Data":"cafe741e683343cf390b6ac9db95403ee5f312e1684cad96266988179395c0b9"} Nov 04 01:15:15.828106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.828023 4486 scope.go:122] "RemoveContainer" containerID="20a245d04c07f5fa4ad99b98e712952ead20b378bb89b378282b63eca1047186" Nov 04 01:15:15.828352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.828334 4486 scope.go:122] "RemoveContainer" containerID="cafe741e683343cf390b6ac9db95403ee5f312e1684cad96266988179395c0b9" Nov 04 01:15:15.828627 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:15.828608 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc_odh-ai-gateway-infra(ba68bb44-72c8-4dc4-8040-3ffd18d9f019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" Nov 04 01:15:15.939620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:15.939577 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:16.342184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.342156 4486 scope.go:122] "RemoveContainer" containerID="05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6" Nov 04 01:15:16.342349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.342240 4486 scope.go:122] "RemoveContainer" containerID="338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac" Nov 04 01:15:16.350845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.350821 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m"] Nov 04 01:15:16.355889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.355872 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.408609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.408567 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m"] Nov 04 01:15:16.409267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.409240 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:16.434630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.434596 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.434807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.434711 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvq62\" (UniqueName: \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.435239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.435063 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.536036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.536005 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.536250 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.536069 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.536250 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.536148 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvq62\" (UniqueName: \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.536672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.536650 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.538483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.538454 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.590564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.590493 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvq62\" (UniqueName: \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\") pod \"maas-api-revoke-sub-ee9685b6-f557m\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.756846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.756813 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:16.857446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.856759 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerStarted","Data":"53563d78c892e8718e981dfe064a13e76e89e8a1f2b6d3e958ef9570a1b5c0ec"} Nov 04 01:15:16.860020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.859778 4486 generic.go:356] "Generic (PLEG): container finished" podID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerID="33018e0889decc6abbe1bc51c21089e302ac3e1ba7ad179d070c6073d2979d28" exitCode=60 Nov 04 01:15:16.860020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.859851 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerDied","Data":"33018e0889decc6abbe1bc51c21089e302ac3e1ba7ad179d070c6073d2979d28"} Nov 04 01:15:16.860020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.859887 4486 scope.go:122] "RemoveContainer" containerID="d2a6e9bb6dafcaba4080274f17ef64808009d5a2ffde11c376773d87a51f3aec" Nov 04 01:15:16.862169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:16.860251 4486 scope.go:122] "RemoveContainer" containerID="33018e0889decc6abbe1bc51c21089e302ac3e1ba7ad179d070c6073d2979d28" Nov 04 01:15:16.862169 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:16.860540 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz_odh-ai-gateway-infra(3da3cf5f-208f-44d5-b2bb-5e4c7ddda982)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" Nov 04 01:15:17.011549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.011522 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:17.132789 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:17.132758 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91083279_ddc7_4738_bf50_088f75a01ab9.slice/crio-c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c WatchSource:0}: Error finding container c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c: Status 404 returned error can't find the container with id c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c Nov 04 01:15:17.133467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.133443 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m"] Nov 04 01:15:17.147090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.147058 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\" (UniqueName: \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\") pod \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " Nov 04 01:15:17.147227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.147125 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\") pod \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\" (UID: \"ba68bb44-72c8-4dc4-8040-3ffd18d9f019\") " Nov 04 01:15:17.147520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.147493 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca" pod "ba68bb44-72c8-4dc4-8040-3ffd18d9f019" (UID: "ba68bb44-72c8-4dc4-8040-3ffd18d9f019"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:17.147700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.147680 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:17.149730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.149709 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n" pod "ba68bb44-72c8-4dc4-8040-3ffd18d9f019" (UID: "ba68bb44-72c8-4dc4-8040-3ffd18d9f019"). InnerVolumeSpecName "kube-api-access-jcg6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:17.248784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.248734 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcg6n\" (UniqueName: \"kubernetes.io/projected/ba68bb44-72c8-4dc4-8040-3ffd18d9f019-kube-api-access-jcg6n\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:17.342286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.342261 4486 scope.go:122] "RemoveContainer" containerID="b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6" Nov 04 01:15:17.868251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.868211 4486 generic.go:356] "Generic (PLEG): container finished" podID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerID="53563d78c892e8718e981dfe064a13e76e89e8a1f2b6d3e958ef9570a1b5c0ec" exitCode=60 Nov 04 01:15:17.868672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.868285 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerDied","Data":"53563d78c892e8718e981dfe064a13e76e89e8a1f2b6d3e958ef9570a1b5c0ec"} Nov 04 01:15:17.868672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.868370 4486 scope.go:122] "RemoveContainer" containerID="05c781035e3cb96b7329529d8b0ea85abb80abd447c7ad8accd87eda7bce13a6" Nov 04 01:15:17.868875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.868744 4486 scope.go:122] "RemoveContainer" containerID="53563d78c892e8718e981dfe064a13e76e89e8a1f2b6d3e958ef9570a1b5c0ec" Nov 04 01:15:17.869101 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:17.869074 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895915-zkkdn_odh-ai-gateway-infra(58c7b28d-82a9-4129-a52e-7cff71dc90f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" Nov 04 01:15:17.871969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.871945 4486 generic.go:356] "Generic (PLEG): container finished" podID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerID="7a5a92ab5b8a368f628c78f77e435945a661b2d3f58149c67416eaeac1a49d54" exitCode=60 Nov 04 01:15:17.872103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.872034 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerDied","Data":"7a5a92ab5b8a368f628c78f77e435945a661b2d3f58149c67416eaeac1a49d54"} Nov 04 01:15:17.872465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.872448 4486 scope.go:122] "RemoveContainer" containerID="7a5a92ab5b8a368f628c78f77e435945a661b2d3f58149c67416eaeac1a49d54" Nov 04 01:15:17.872716 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:17.872696 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj_odh-ai-gateway-infra(77de7f8b-1f5b-4edc-b7f1-e06914f72c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" Nov 04 01:15:17.873386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.873370 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" Nov 04 01:15:17.873386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.873375 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc" event={"ID":"ba68bb44-72c8-4dc4-8040-3ffd18d9f019","Type":"ContainerDied","Data":"93159679eab0d810dcef2a5daf22471b87f59c479c6ca431a53acae48d15119c"} Nov 04 01:15:17.875452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.875396 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerStarted","Data":"0b301894daff0cb2bd41c253c1106ed5918a993339d55ed6086e1bd43a995b46"} Nov 04 01:15:17.878329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.878306 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" event={"ID":"91083279-ddc7-4738-bf50-088f75a01ab9","Type":"ContainerStarted","Data":"0522dd12c2c122b40d27be1dee7985e46861026926e8d923cdf0db6098335856"} Nov 04 01:15:17.878422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.878348 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" event={"ID":"91083279-ddc7-4738-bf50-088f75a01ab9","Type":"ContainerStarted","Data":"c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c"} Nov 04 01:15:17.881867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.881851 4486 scope.go:122] "RemoveContainer" containerID="338911bf868a34d8ff0f84cefd484a963ac7f14b88710ab3b13cb2eb79a4a7ac" Nov 04 01:15:17.894508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:17.894490 4486 scope.go:122] "RemoveContainer" containerID="cafe741e683343cf390b6ac9db95403ee5f312e1684cad96266988179395c0b9" Nov 04 01:15:18.083389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.083353 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:18.088278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.088242 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:18.092022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.091997 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895915-274cc"] Nov 04 01:15:18.200142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.200045 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:18.202189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.202141 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" podStartSLOduration=2.20212693 podStartE2EDuration="2.20212693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:18.198307084 +0000 UTC m=+1086.956969244" watchObservedRunningTime="2026-11-04 01:15:18.20212693 +0000 UTC m=+1086.960789088" Nov 04 01:15:18.341762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.341723 4486 scope.go:122] "RemoveContainer" containerID="99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841" Nov 04 01:15:18.366219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.366196 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:18.463402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.463302 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\" (UniqueName: \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\") pod \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " Nov 04 01:15:18.463582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.463398 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\") pod \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\" (UID: \"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982\") " Nov 04 01:15:18.463776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.463736 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca" pod "3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" (UID: "3da3cf5f-208f-44d5-b2bb-5e4c7ddda982"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:18.464093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.464063 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:18.465626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.465604 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4" pod "3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" (UID: "3da3cf5f-208f-44d5-b2bb-5e4c7ddda982"). InnerVolumeSpecName "kube-api-access-4z7g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:18.565280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.565247 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7g4\" (UniqueName: \"kubernetes.io/projected/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982-kube-api-access-4z7g4\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:18.889180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.889142 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerStarted","Data":"a9aa65e574b5c6fac9305c9ef65042437e1488081439f8a63cec534aa8c104d4"} Nov 04 01:15:18.890614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.890591 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" Nov 04 01:15:18.890970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.890613 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz" event={"ID":"3da3cf5f-208f-44d5-b2bb-5e4c7ddda982","Type":"ContainerDied","Data":"db18a935a60a14c0d8348ea1b159b9c64b8a291829e0695577810bc609fcfc75"} Nov 04 01:15:18.890970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.890697 4486 scope.go:122] "RemoveContainer" containerID="33018e0889decc6abbe1bc51c21089e302ac3e1ba7ad179d070c6073d2979d28" Nov 04 01:15:18.897037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.896110 4486 generic.go:356] "Generic (PLEG): container finished" podID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerID="0b301894daff0cb2bd41c253c1106ed5918a993339d55ed6086e1bd43a995b46" exitCode=60 Nov 04 01:15:18.897037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.896205 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerDied","Data":"0b301894daff0cb2bd41c253c1106ed5918a993339d55ed6086e1bd43a995b46"} Nov 04 01:15:18.897037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.896619 4486 scope.go:122] "RemoveContainer" containerID="0b301894daff0cb2bd41c253c1106ed5918a993339d55ed6086e1bd43a995b46" Nov 04 01:15:18.897037 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:18.896918 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g_odh-ai-gateway-infra(77d94d58-ae32-4257-b0de-d545c4ab9f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" Nov 04 01:15:18.899418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.899389 4486 generic.go:356] "Generic (PLEG): container finished" podID="91083279-ddc7-4738-bf50-088f75a01ab9" containerID="0522dd12c2c122b40d27be1dee7985e46861026926e8d923cdf0db6098335856" exitCode=0 Nov 04 01:15:18.899520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.899469 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" event={"ID":"91083279-ddc7-4738-bf50-088f75a01ab9","Type":"ContainerDied","Data":"0522dd12c2c122b40d27be1dee7985e46861026926e8d923cdf0db6098335856"} Nov 04 01:15:18.906948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:18.906926 4486 scope.go:122] "RemoveContainer" containerID="b1d27a0910789b4c641151c314a0fc9786a8a4ed80be86630c4b5cb417bdeac6" Nov 04 01:15:19.061310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.061285 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:19.171775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.171739 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\" (UniqueName: \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\") pod \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " Nov 04 01:15:19.171960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.171826 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\") pod \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\" (UID: \"58c7b28d-82a9-4129-a52e-7cff71dc90f0\") " Nov 04 01:15:19.172319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.172273 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca" pod "58c7b28d-82a9-4129-a52e-7cff71dc90f0" (UID: "58c7b28d-82a9-4129-a52e-7cff71dc90f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:19.174054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.174007 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5" pod "58c7b28d-82a9-4129-a52e-7cff71dc90f0" (UID: "58c7b28d-82a9-4129-a52e-7cff71dc90f0"). InnerVolumeSpecName "kube-api-access-kgkk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:19.215871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.215841 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:19.226073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.226044 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895915-5kzjz"] Nov 04 01:15:19.273224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.273186 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkk5\" (UniqueName: \"kubernetes.io/projected/58c7b28d-82a9-4129-a52e-7cff71dc90f0-kube-api-access-kgkk5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:19.273224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.273221 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c7b28d-82a9-4129-a52e-7cff71dc90f0-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:19.316646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.316615 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:19.352850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.352822 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" path="/var/lib/kubelet/pods/3da3cf5f-208f-44d5-b2bb-5e4c7ddda982/volumes" Nov 04 01:15:19.353303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.353283 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" path="/var/lib/kubelet/pods/ba68bb44-72c8-4dc4-8040-3ffd18d9f019/volumes" Nov 04 01:15:19.429587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.429553 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:19.610069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.610043 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:19.677629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.677603 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\" (UniqueName: \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\") pod \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " Nov 04 01:15:19.677785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.677642 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\") pod \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\" (UID: \"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d\") " Nov 04 01:15:19.678047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.678016 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca" pod "77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" (UID: "77de7f8b-1f5b-4edc-b7f1-e06914f72c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:19.678191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.678113 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:19.679693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.679669 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm" pod "77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" (UID: "77de7f8b-1f5b-4edc-b7f1-e06914f72c1d"). InnerVolumeSpecName "kube-api-access-hzsfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:19.779749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.779717 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzsfm\" (UniqueName: \"kubernetes.io/projected/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d-kube-api-access-hzsfm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:19.918390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.918367 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" Nov 04 01:15:19.918770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.918369 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn" event={"ID":"58c7b28d-82a9-4129-a52e-7cff71dc90f0","Type":"ContainerDied","Data":"ddacee379af3bc351d5855a08ea505f17cef58d7e3af42ffcaed23c04aa915c9"} Nov 04 01:15:19.918770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.918559 4486 scope.go:122] "RemoveContainer" containerID="53563d78c892e8718e981dfe064a13e76e89e8a1f2b6d3e958ef9570a1b5c0ec" Nov 04 01:15:19.920559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.920530 4486 generic.go:356] "Generic (PLEG): container finished" podID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerID="a9aa65e574b5c6fac9305c9ef65042437e1488081439f8a63cec534aa8c104d4" exitCode=60 Nov 04 01:15:19.920675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.920663 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerDied","Data":"a9aa65e574b5c6fac9305c9ef65042437e1488081439f8a63cec534aa8c104d4"} Nov 04 01:15:19.921111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.921091 4486 scope.go:122] "RemoveContainer" containerID="a9aa65e574b5c6fac9305c9ef65042437e1488081439f8a63cec534aa8c104d4" Nov 04 01:15:19.921517 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:19.921488 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728_odh-ai-gateway-infra(0150afb8-84e5-4cd1-a8ae-e3fc691518e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" Nov 04 01:15:19.923468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.923452 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" Nov 04 01:15:19.923577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.923493 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj" event={"ID":"77de7f8b-1f5b-4edc-b7f1-e06914f72c1d","Type":"ContainerDied","Data":"cdd1e12cfb06743684584680a265eedda295036bea9a2db72e743194aa86d4c9"} Nov 04 01:15:19.929448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.929415 4486 scope.go:122] "RemoveContainer" containerID="99ea7c0147c212c9255487f3fe13ed0100dac72f1632c2a039643eba0b18c841" Nov 04 01:15:19.964048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:19.962865 4486 scope.go:122] "RemoveContainer" containerID="7a5a92ab5b8a368f628c78f77e435945a661b2d3f58149c67416eaeac1a49d54" Nov 04 01:15:20.104669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.104646 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:20.111757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.111729 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:20.134788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.134766 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:20.138205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.138184 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c563ee-29895915-dkvsj"] Nov 04 01:15:20.183779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.183724 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\" (UniqueName: \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\") pod \"77d94d58-ae32-4257-b0de-d545c4ab9f40\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " Nov 04 01:15:20.183779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.183776 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\") pod \"77d94d58-ae32-4257-b0de-d545c4ab9f40\" (UID: \"77d94d58-ae32-4257-b0de-d545c4ab9f40\") " Nov 04 01:15:20.183927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.183870 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\") pod \"91083279-ddc7-4738-bf50-088f75a01ab9\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " Nov 04 01:15:20.183927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.183892 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\" (UniqueName: \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\") pod \"91083279-ddc7-4738-bf50-088f75a01ab9\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " Nov 04 01:15:20.183927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.183916 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\") pod \"91083279-ddc7-4738-bf50-088f75a01ab9\" (UID: \"91083279-ddc7-4738-bf50-088f75a01ab9\") " Nov 04 01:15:20.184343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.184309 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca" pod "77d94d58-ae32-4257-b0de-d545c4ab9f40" (UID: "77d94d58-ae32-4257-b0de-d545c4ab9f40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:20.184448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.184387 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca" pod "91083279-ddc7-4738-bf50-088f75a01ab9" (UID: "91083279-ddc7-4738-bf50-088f75a01ab9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:20.185919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.185895 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl" pod "77d94d58-ae32-4257-b0de-d545c4ab9f40" (UID: "77d94d58-ae32-4257-b0de-d545c4ab9f40"). InnerVolumeSpecName "kube-api-access-47jsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:20.186018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.185924 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca" pod "91083279-ddc7-4738-bf50-088f75a01ab9" (UID: "91083279-ddc7-4738-bf50-088f75a01ab9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:20.186065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.186017 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62" pod "91083279-ddc7-4738-bf50-088f75a01ab9" (UID: "91083279-ddc7-4738-bf50-088f75a01ab9"). InnerVolumeSpecName "kube-api-access-zvq62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:20.244420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.244392 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:20.249624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.249599 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895915-zkkdn"] Nov 04 01:15:20.285503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.285478 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jsl\" (UniqueName: \"kubernetes.io/projected/77d94d58-ae32-4257-b0de-d545c4ab9f40-kube-api-access-47jsl\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:20.285503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.285501 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d94d58-ae32-4257-b0de-d545c4ab9f40-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:20.285634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.285514 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:20.285634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.285523 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvq62\" (UniqueName: \"kubernetes.io/projected/91083279-ddc7-4738-bf50-088f75a01ab9-kube-api-access-zvq62\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:20.285634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.285532 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91083279-ddc7-4738-bf50-088f75a01ab9-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:20.342961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.342932 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:20.931416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.931330 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" Nov 04 01:15:20.931416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.931328 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g" event={"ID":"77d94d58-ae32-4257-b0de-d545c4ab9f40","Type":"ContainerDied","Data":"e34403b155c8d0251621a14fef47ac92db68de1c296a1944373b407ef2fd859b"} Nov 04 01:15:20.931932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.931485 4486 scope.go:122] "RemoveContainer" containerID="0b301894daff0cb2bd41c253c1106ed5918a993339d55ed6086e1bd43a995b46" Nov 04 01:15:20.933131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.933114 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" Nov 04 01:15:20.933255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.933231 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m" event={"ID":"91083279-ddc7-4738-bf50-088f75a01ab9","Type":"ContainerDied","Data":"c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c"} Nov 04 01:15:20.933319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:20.933280 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78568faddf3ddb79449d34dc186dd4eebe870c54f71fa925956570dd220324c" Nov 04 01:15:21.092379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.092353 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:21.195100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.195016 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\" (UniqueName: \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\") pod \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " Nov 04 01:15:21.195256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.195153 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\") pod \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\" (UID: \"0150afb8-84e5-4cd1-a8ae-e3fc691518e9\") " Nov 04 01:15:21.195578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.195549 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca" pod "0150afb8-84e5-4cd1-a8ae-e3fc691518e9" (UID: "0150afb8-84e5-4cd1-a8ae-e3fc691518e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:21.195708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.195691 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:21.197229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.197199 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l" pod "0150afb8-84e5-4cd1-a8ae-e3fc691518e9" (UID: "0150afb8-84e5-4cd1-a8ae-e3fc691518e9"). InnerVolumeSpecName "kube-api-access-9pj4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:21.197689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.197669 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:21.201169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.201148 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2ee045-29895915-72b7g"] Nov 04 01:15:21.296586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.296552 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pj4l\" (UniqueName: \"kubernetes.io/projected/0150afb8-84e5-4cd1-a8ae-e3fc691518e9-kube-api-access-9pj4l\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:21.351389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.351348 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" path="/var/lib/kubelet/pods/58c7b28d-82a9-4129-a52e-7cff71dc90f0/volumes" Nov 04 01:15:21.352021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.351995 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" path="/var/lib/kubelet/pods/77d94d58-ae32-4257-b0de-d545c4ab9f40/volumes" Nov 04 01:15:21.352577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.352559 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" path="/var/lib/kubelet/pods/77de7f8b-1f5b-4edc-b7f1-e06914f72c1d/volumes" Nov 04 01:15:21.951813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.951776 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" event={"ID":"0150afb8-84e5-4cd1-a8ae-e3fc691518e9","Type":"ContainerDied","Data":"0600fd0fa125dd242c916e28ec157c5eeef4f52a7ace8f799641b107d33d0ff2"} Nov 04 01:15:21.952360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.951848 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728" Nov 04 01:15:21.952360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:21.951864 4486 scope.go:122] "RemoveContainer" containerID="a9aa65e574b5c6fac9305c9ef65042437e1488081439f8a63cec534aa8c104d4" Nov 04 01:15:22.121743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:22.116506 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:22.121932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:22.121903 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e65b23-29895915-wg728"] Nov 04 01:15:23.361009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:23.360276 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" path="/var/lib/kubelet/pods/0150afb8-84e5-4cd1-a8ae-e3fc691518e9/volumes" Nov 04 01:15:31.547960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.547926 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29"] Nov 04 01:15:31.548749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548727 4486 cpu_manager.go:414] "containerMap: removing container" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548752 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548767 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.548799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548775 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.548799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548788 4486 cpu_manager.go:414] "containerMap: removing container" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548795 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548807 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548814 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548823 4486 cpu_manager.go:414] "containerMap: removing container" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548830 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548862 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548869 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548881 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548889 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548900 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548907 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548916 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548922 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.548939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548941 4486 cpu_manager.go:414] "containerMap: removing container" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548948 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548973 4486 cpu_manager.go:414] "containerMap: removing container" podUID="91083279-ddc7-4738-bf50-088f75a01ab9" containerName="revoke-keys" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.548995 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91083279-ddc7-4738-bf50-088f75a01ab9" containerName="revoke-keys" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549008 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549014 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549026 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549032 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549041 4486 cpu_manager.go:414] "containerMap: removing container" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549048 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549056 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549063 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549199 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549213 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549226 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549235 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="91083279-ddc7-4738-bf50-088f75a01ab9" containerName="revoke-keys" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549246 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549255 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549262 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549269 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549281 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549290 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549299 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549306 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549316 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c7b28d-82a9-4129-a52e-7cff71dc90f0" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549325 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549337 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77de7f8b-1f5b-4edc-b7f1-e06914f72c1d" containerName="cleanup" Nov 04 01:15:31.549340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.549346 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:31.553500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.553477 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.603317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.603287 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29"] Nov 04 01:15:31.646107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.646082 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:31.646263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.646142 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:15:31.662727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.662698 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frsnk\" (UniqueName: \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.662881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.662778 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.662946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.662894 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.763651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.763608 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frsnk\" (UniqueName: \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.763835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.763695 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.763835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.763756 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.764520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.764496 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.777025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.766404 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.817677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.817644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frsnk\" (UniqueName: \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\") pod \"maas-api-revoke-sub-bf40af40-k2z29\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:31.869451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:31.869411 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:32.241375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.241341 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29"] Nov 04 01:15:32.242838 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:32.242806 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f2e88d_a3a0_45e7_a3c1_aba4eb6ff884.slice/crio-7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d WatchSource:0}: Error finding container 7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d: Status 404 returned error can't find the container with id 7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d Nov 04 01:15:32.308891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.308861 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz"] Nov 04 01:15:32.309402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309388 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:32.309451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309405 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:32.309451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309421 4486 cpu_manager.go:414] "containerMap: removing container" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:32.309451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309426 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba68bb44-72c8-4dc4-8040-3ffd18d9f019" containerName="cleanup" Nov 04 01:15:32.309451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309449 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:32.309577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309456 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:32.309577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309477 4486 cpu_manager.go:414] "containerMap: removing container" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:32.309577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309485 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:32.309671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309661 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da3cf5f-208f-44d5-b2bb-5e4c7ddda982" containerName="cleanup" Nov 04 01:15:32.309706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309675 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d94d58-ae32-4257-b0de-d545c4ab9f40" containerName="cleanup" Nov 04 01:15:32.309706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.309683 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0150afb8-84e5-4cd1-a8ae-e3fc691518e9" containerName="cleanup" Nov 04 01:15:32.314457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.314441 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.366312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.366284 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz"] Nov 04 01:15:32.472948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.472917 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpt7\" (UniqueName: \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.473107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.472966 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.473176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.473122 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.574095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.574070 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpt7\" (UniqueName: \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.574493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.574125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.574493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.574256 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.574900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.574874 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.576370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.576349 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.628078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.628045 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpt7\" (UniqueName: \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\") pod \"maas-api-revoke-sub-d624158f-4frkz\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:32.927082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:32.927051 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:33.071037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:33.070979 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" event={"ID":"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884","Type":"ContainerStarted","Data":"81b07f3e6c049a9c33509af0575388adaf29d4c02334dc2d7a37bc37062f2fa5"} Nov 04 01:15:33.071223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:33.071065 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" event={"ID":"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884","Type":"ContainerStarted","Data":"7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d"} Nov 04 01:15:33.181554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:33.181464 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" podStartSLOduration=2.181449715 podStartE2EDuration="2.181449715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:33.179578425 +0000 UTC m=+1101.938240588" watchObservedRunningTime="2026-11-04 01:15:33.181449715 +0000 UTC m=+1101.940111876" Nov 04 01:15:33.298911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:33.298882 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz"] Nov 04 01:15:33.299657 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:33.299633 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9641c0ff_fce2_40a1_9c76_efebfb61319a.slice/crio-05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a WatchSource:0}: Error finding container 05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a: Status 404 returned error can't find the container with id 05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a Nov 04 01:15:34.079706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.079668 4486 generic.go:356] "Generic (PLEG): container finished" podID="60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" containerID="81b07f3e6c049a9c33509af0575388adaf29d4c02334dc2d7a37bc37062f2fa5" exitCode=0 Nov 04 01:15:34.080152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.079757 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" event={"ID":"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884","Type":"ContainerDied","Data":"81b07f3e6c049a9c33509af0575388adaf29d4c02334dc2d7a37bc37062f2fa5"} Nov 04 01:15:34.081610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.081587 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" event={"ID":"9641c0ff-fce2-40a1-9c76-efebfb61319a","Type":"ContainerStarted","Data":"f1954d9f74033f2568e7fc1ad4c11be6486942a81932cfb28eb5abbf5caffc09"} Nov 04 01:15:34.081723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.081630 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" event={"ID":"9641c0ff-fce2-40a1-9c76-efebfb61319a","Type":"ContainerStarted","Data":"05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a"} Nov 04 01:15:34.181890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.181853 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j"] Nov 04 01:15:34.185794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.185771 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.201373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.201345 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j"] Nov 04 01:15:34.292282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.292253 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.292449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.292388 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjddq\" (UniqueName: \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.292449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.292425 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.298557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.298510 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" podStartSLOduration=2.298496337 podStartE2EDuration="2.298496337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:34.29520843 +0000 UTC m=+1103.053870590" watchObservedRunningTime="2026-11-04 01:15:34.298496337 +0000 UTC m=+1103.057158498" Nov 04 01:15:34.393772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.393699 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.393931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.393777 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjddq\" (UniqueName: \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.393931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.393798 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.394436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.394417 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.395893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.395875 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.448616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.448584 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjddq\" (UniqueName: \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\") pod \"maas-api-revoke-sub-8cca7975-9tw8j\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.505305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.505267 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:34.874715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:34.874687 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j"] Nov 04 01:15:34.874897 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:34.874842 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119fa33a_8d9f_4d4d_a325_2ae21c5c5607.slice/crio-1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3 WatchSource:0}: Error finding container 1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3: Status 404 returned error can't find the container with id 1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3 Nov 04 01:15:35.089665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.089633 4486 generic.go:356] "Generic (PLEG): container finished" podID="9641c0ff-fce2-40a1-9c76-efebfb61319a" containerID="f1954d9f74033f2568e7fc1ad4c11be6486942a81932cfb28eb5abbf5caffc09" exitCode=0 Nov 04 01:15:35.090102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.089677 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" event={"ID":"9641c0ff-fce2-40a1-9c76-efebfb61319a","Type":"ContainerDied","Data":"f1954d9f74033f2568e7fc1ad4c11be6486942a81932cfb28eb5abbf5caffc09"} Nov 04 01:15:35.091039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.091019 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" event={"ID":"119fa33a-8d9f-4d4d-a325-2ae21c5c5607","Type":"ContainerStarted","Data":"1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3"} Nov 04 01:15:35.247140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.247114 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:35.406751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.406667 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\") pod \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " Nov 04 01:15:35.406897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.406812 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\" (UniqueName: \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\") pod \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " Nov 04 01:15:35.406897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.406845 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\") pod \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\" (UID: \"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884\") " Nov 04 01:15:35.407264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.407237 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca" pod "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" (UID: "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:35.408709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.408679 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca" pod "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" (UID: "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:35.408811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.408743 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk" pod "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" (UID: "60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884"). InnerVolumeSpecName "kube-api-access-frsnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:35.507694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.507663 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frsnk\" (UniqueName: \"kubernetes.io/projected/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-kube-api-access-frsnk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:35.507694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.507692 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:35.507878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:35.507708 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:36.101360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.101326 4486 generic.go:356] "Generic (PLEG): container finished" podID="119fa33a-8d9f-4d4d-a325-2ae21c5c5607" containerID="dd997a8a18c73ca8d8839384c2cd8e8c665989d1cf6fa23167da195913382f90" exitCode=0 Nov 04 01:15:36.101760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.101410 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" event={"ID":"119fa33a-8d9f-4d4d-a325-2ae21c5c5607","Type":"ContainerDied","Data":"dd997a8a18c73ca8d8839384c2cd8e8c665989d1cf6fa23167da195913382f90"} Nov 04 01:15:36.102926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.102906 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" Nov 04 01:15:36.103124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.102938 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29" event={"ID":"60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884","Type":"ContainerDied","Data":"7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d"} Nov 04 01:15:36.103124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.102976 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7087d447a92f6a53eff88e2dcff9c0fbd887e57262784c4067ee56531828945d" Nov 04 01:15:36.261174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.261145 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:36.420856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.420784 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\") pod \"9641c0ff-fce2-40a1-9c76-efebfb61319a\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " Nov 04 01:15:36.421009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.420878 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\") pod \"9641c0ff-fce2-40a1-9c76-efebfb61319a\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " Nov 04 01:15:36.421009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.420913 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\" (UniqueName: \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\") pod \"9641c0ff-fce2-40a1-9c76-efebfb61319a\" (UID: \"9641c0ff-fce2-40a1-9c76-efebfb61319a\") " Nov 04 01:15:36.421304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.421265 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca" pod "9641c0ff-fce2-40a1-9c76-efebfb61319a" (UID: "9641c0ff-fce2-40a1-9c76-efebfb61319a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:36.421430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.421347 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:36.422780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.422759 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca" pod "9641c0ff-fce2-40a1-9c76-efebfb61319a" (UID: "9641c0ff-fce2-40a1-9c76-efebfb61319a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:36.423008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.422975 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7" pod "9641c0ff-fce2-40a1-9c76-efebfb61319a" (UID: "9641c0ff-fce2-40a1-9c76-efebfb61319a"). InnerVolumeSpecName "kube-api-access-9wpt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:36.524707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.524668 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9641c0ff-fce2-40a1-9c76-efebfb61319a-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:36.524707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:36.524708 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpt7\" (UniqueName: \"kubernetes.io/projected/9641c0ff-fce2-40a1-9c76-efebfb61319a-kube-api-access-9wpt7\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:37.112011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.111938 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" event={"ID":"9641c0ff-fce2-40a1-9c76-efebfb61319a","Type":"ContainerDied","Data":"05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a"} Nov 04 01:15:37.112413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.112041 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05592711726c3a67a5b3ba6481f46500e47977c62f4980114d8c682b412fec0a" Nov 04 01:15:37.112413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.111953 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz" Nov 04 01:15:37.266080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.266056 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:37.439436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.439337 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\" (UniqueName: \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\") pod \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " Nov 04 01:15:37.439436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.439418 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\") pod \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " Nov 04 01:15:37.439661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.439462 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\") pod \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\" (UID: \"119fa33a-8d9f-4d4d-a325-2ae21c5c5607\") " Nov 04 01:15:37.440905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.440874 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca" pod "119fa33a-8d9f-4d4d-a325-2ae21c5c5607" (UID: "119fa33a-8d9f-4d4d-a325-2ae21c5c5607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:37.443051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.443018 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq" pod "119fa33a-8d9f-4d4d-a325-2ae21c5c5607" (UID: "119fa33a-8d9f-4d4d-a325-2ae21c5c5607"). InnerVolumeSpecName "kube-api-access-zjddq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:37.444360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.444329 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca" pod "119fa33a-8d9f-4d4d-a325-2ae21c5c5607" (UID: "119fa33a-8d9f-4d4d-a325-2ae21c5c5607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:37.540955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.540925 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjddq\" (UniqueName: \"kubernetes.io/projected/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-kube-api-access-zjddq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:37.540955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.540951 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:37.541148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:37.540962 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/119fa33a-8d9f-4d4d-a325-2ae21c5c5607-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:38.120489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:38.120454 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" Nov 04 01:15:38.120908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:38.120479 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j" event={"ID":"119fa33a-8d9f-4d4d-a325-2ae21c5c5607","Type":"ContainerDied","Data":"1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3"} Nov 04 01:15:38.120908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:38.120569 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb57ec14e1c8cbb85182b480943b6dea7f7b78033d532d5498f9ccb562035e3" Nov 04 01:15:40.278448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.278418 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:15:40.278970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.278955 4486 cpu_manager.go:414] "containerMap: removing container" podUID="119fa33a-8d9f-4d4d-a325-2ae21c5c5607" containerName="revoke-keys" Nov 04 01:15:40.279055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.278971 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119fa33a-8d9f-4d4d-a325-2ae21c5c5607" containerName="revoke-keys" Nov 04 01:15:40.279055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279005 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9641c0ff-fce2-40a1-9c76-efebfb61319a" containerName="revoke-keys" Nov 04 01:15:40.279055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279013 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9641c0ff-fce2-40a1-9c76-efebfb61319a" containerName="revoke-keys" Nov 04 01:15:40.279055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279033 4486 cpu_manager.go:414] "containerMap: removing container" podUID="60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" containerName="revoke-keys" Nov 04 01:15:40.279055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279039 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" containerName="revoke-keys" Nov 04 01:15:40.279210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279145 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="119fa33a-8d9f-4d4d-a325-2ae21c5c5607" containerName="revoke-keys" Nov 04 01:15:40.279210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279156 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9641c0ff-fce2-40a1-9c76-efebfb61319a" containerName="revoke-keys" Nov 04 01:15:40.279210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.279163 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" containerName="revoke-keys" Nov 04 01:15:40.282978 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.282959 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370237 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370357 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370434 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj92x\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370462 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370493 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370629 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370767 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370857 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.370947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.370888 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.472435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.472401 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:15:40.473010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.472970 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473050 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj92x\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473156 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473178 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473208 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473284 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473345 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473384 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473478 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.473739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.473604 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.474216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.474191 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.474371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.474266 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.474454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.474327 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.476557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.476529 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.476917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.476898 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.528962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.528906 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.529358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.529337 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj92x\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\") pod \"maas-default-gateway-openshift-default-694db55c87-5hlrs\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:40.596083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:40.596053 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:41.016835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.016803 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:15:41.017416 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:41.017388 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409db713_1d4e_4c3d_8f69_8c402a4f52ec.slice/crio-0075bd35496714222e9695bfe2daf9c67eef8ed75d7347e8455bfb59b3a4e5e6 WatchSource:0}: Error finding container 0075bd35496714222e9695bfe2daf9c67eef8ed75d7347e8455bfb59b3a4e5e6: Status 404 returned error can't find the container with id 0075bd35496714222e9695bfe2daf9c67eef8ed75d7347e8455bfb59b3a4e5e6 Nov 04 01:15:41.019675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.019643 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:41.019757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.019708 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:41.019757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.019740 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:41.144040 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.144009 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" event={"ID":"409db713-1d4e-4c3d-8f69-8c402a4f52ec","Type":"ContainerStarted","Data":"f54397dd35fb6ca1c3ca098033ae567347b5b97d2ca12b77cf04a450b9366837"} Nov 04 01:15:41.144209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.144072 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" event={"ID":"409db713-1d4e-4c3d-8f69-8c402a4f52ec","Type":"ContainerStarted","Data":"0075bd35496714222e9695bfe2daf9c67eef8ed75d7347e8455bfb59b3a4e5e6"} Nov 04 01:15:41.259676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.259620 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podStartSLOduration=1.259605172 podStartE2EDuration="1.259605172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:41.257040292 +0000 UTC m=+1110.015702455" watchObservedRunningTime="2026-11-04 01:15:41.259605172 +0000 UTC m=+1110.018267374" Nov 04 01:15:41.596562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:41.596530 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:42.598184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:42.598144 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:42.598592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:42.598207 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:43.596963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:43.596919 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:43.597168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:43.597031 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:43.853850 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:15:43.853690 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:15:44.596965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:44.596919 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:44.597170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:44.596974 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:45.455583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.455546 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz"] Nov 04 01:15:45.459825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.459805 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.494808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.494785 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz"] Nov 04 01:15:45.509292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.509267 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:45.553199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.553174 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:15:45.555387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.554551 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.555559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.555538 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6z9\" (UniqueName: \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.555661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.555642 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.596370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.596336 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:45.596503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.596390 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:45.656404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.656359 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6z9\" (UniqueName: \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.656590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.656478 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.656590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.656537 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.665020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.657697 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.665590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.665565 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.710205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.710140 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6z9\" (UniqueName: \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\") pod \"maas-api-revoke-sub-17e6de66-fblnz\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:45.774047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:45.774013 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:46.157822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.157019 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:15:46.163253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.163229 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.175788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.175075 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz"] Nov 04 01:15:46.199349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.199324 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" event={"ID":"7147e754-f7ab-4b97-91da-6a9df9b6cb06","Type":"ContainerStarted","Data":"81d68521643e537680089bfa7625b698ec6e09dad7e1e2195d6fcda2c12c8631"} Nov 04 01:15:46.213846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.213827 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-678aaf-openshift-default-dockercfg-88gm8\"" Nov 04 01:15:46.260005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.259380 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:15:46.264732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264698 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.264850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264789 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbvmq\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.264850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264818 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.264850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264842 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.264946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264870 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.264946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264901 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.265038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.264945 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.265075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.265037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.265108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.265071 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366392 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366462 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbvmq\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366485 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366499 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366516 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366537 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366568 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366603 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366628 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.366917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.366746 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.367149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.367130 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.367289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.367271 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.367654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.367636 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.367816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.367801 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.368881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.368852 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.369531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.369513 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.421121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.421066 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.421328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.421308 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbvmq\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.495953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.495922 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:46.597434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.597401 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:46.597533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.597465 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:46.951263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.951233 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:15:46.960238 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:46.954351 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb106dfc4_ef90_4c74_a481_91c119cf4c4c.slice/crio-2b72cdedfe1f4955e5e1cf6d65fe6ecc5719972bd8e9169c03dadd3bf10b17c7 WatchSource:0}: Error finding container 2b72cdedfe1f4955e5e1cf6d65fe6ecc5719972bd8e9169c03dadd3bf10b17c7: Status 404 returned error can't find the container with id 2b72cdedfe1f4955e5e1cf6d65fe6ecc5719972bd8e9169c03dadd3bf10b17c7 Nov 04 01:15:46.960238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.957777 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:46.960238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.957863 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:46.960238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:46.957909 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:47.210385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.210348 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" event={"ID":"b106dfc4-ef90-4c74-a481-91c119cf4c4c","Type":"ContainerStarted","Data":"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a"} Nov 04 01:15:47.210509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.210424 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" event={"ID":"b106dfc4-ef90-4c74-a481-91c119cf4c4c","Type":"ContainerStarted","Data":"2b72cdedfe1f4955e5e1cf6d65fe6ecc5719972bd8e9169c03dadd3bf10b17c7"} Nov 04 01:15:47.212337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.212313 4486 generic.go:356] "Generic (PLEG): container finished" podID="7147e754-f7ab-4b97-91da-6a9df9b6cb06" containerID="432b11040222a3b108bbd90a827935d712e5b0fc5a33d9692ca89c6ec4ed5a7f" exitCode=0 Nov 04 01:15:47.212434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.212352 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" event={"ID":"7147e754-f7ab-4b97-91da-6a9df9b6cb06","Type":"ContainerDied","Data":"432b11040222a3b108bbd90a827935d712e5b0fc5a33d9692ca89c6ec4ed5a7f"} Nov 04 01:15:47.326592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.326529 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podStartSLOduration=1.32650986 podStartE2EDuration="1.32650986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:47.324334389 +0000 UTC m=+1116.082996551" watchObservedRunningTime="2026-11-04 01:15:47.32650986 +0000 UTC m=+1116.085172022" Nov 04 01:15:47.496907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.496823 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:47.597325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.597290 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:47.597472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:47.597346 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:48.461267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.461224 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97"] Nov 04 01:15:48.465205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.465179 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:48.466068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.466050 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.494850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.494819 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97"] Nov 04 01:15:48.502919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.500131 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:15:48.502919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.500185 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 04 01:15:48.596842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.596808 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-5hlrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 04 01:15:48.597019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.596864 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 04 01:15:48.599282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599260 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\") pod \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " Nov 04 01:15:48.599389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599348 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\" (UniqueName: \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\") pod \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " Nov 04 01:15:48.599426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599387 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\") pod \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\" (UID: \"7147e754-f7ab-4b97-91da-6a9df9b6cb06\") " Nov 04 01:15:48.599548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599529 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.599617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599602 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghd2r\" (UniqueName: \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.599799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599776 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.599865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599805 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca" pod "7147e754-f7ab-4b97-91da-6a9df9b6cb06" (UID: "7147e754-f7ab-4b97-91da-6a9df9b6cb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:48.599967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.599953 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:48.601941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.601912 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca" pod "7147e754-f7ab-4b97-91da-6a9df9b6cb06" (UID: "7147e754-f7ab-4b97-91da-6a9df9b6cb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:48.602062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.602043 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9" pod "7147e754-f7ab-4b97-91da-6a9df9b6cb06" (UID: "7147e754-f7ab-4b97-91da-6a9df9b6cb06"). InnerVolumeSpecName "kube-api-access-fg6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:48.700487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.700455 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghd2r\" (UniqueName: \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.700658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.700522 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.700658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.700652 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.700790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.700761 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7147e754-f7ab-4b97-91da-6a9df9b6cb06-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:48.700790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.700781 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6z9\" (UniqueName: \"kubernetes.io/projected/7147e754-f7ab-4b97-91da-6a9df9b6cb06-kube-api-access-fg6z9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:48.701322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.701298 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.702900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.702873 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.761708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.761646 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghd2r\" (UniqueName: \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\") pod \"maas-api-revoke-sub-655f832e-rwf97\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.780419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.780391 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:48.870660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.866380 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs"] Nov 04 01:15:48.870660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.867272 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7147e754-f7ab-4b97-91da-6a9df9b6cb06" containerName="revoke-keys" Nov 04 01:15:48.870660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.867295 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7147e754-f7ab-4b97-91da-6a9df9b6cb06" containerName="revoke-keys" Nov 04 01:15:48.870660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.867470 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7147e754-f7ab-4b97-91da-6a9df9b6cb06" containerName="revoke-keys" Nov 04 01:15:48.884003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.878050 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:48.901437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:48.901031 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs"] Nov 04 01:15:49.004853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.004815 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.005187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.005167 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.005354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.005340 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfht\" (UniqueName: \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.106382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.106347 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.106588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.106414 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.106588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.106446 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfht\" (UniqueName: \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.107424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.107403 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.109164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.109147 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.164718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.164687 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfht\" (UniqueName: \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\") pod \"maas-api-revoke-sub-a30128f0-4n4hs\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.211846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.211757 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97"] Nov 04 01:15:49.215462 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:49.215437 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf97e290_05ba_490e_834c_d34e195b76b3.slice/crio-c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f WatchSource:0}: Error finding container c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f: Status 404 returned error can't find the container with id c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f Nov 04 01:15:49.241003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.228120 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" Nov 04 01:15:49.241003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.228261 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz" event={"ID":"7147e754-f7ab-4b97-91da-6a9df9b6cb06","Type":"ContainerDied","Data":"81d68521643e537680089bfa7625b698ec6e09dad7e1e2195d6fcda2c12c8631"} Nov 04 01:15:49.241003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.228306 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d68521643e537680089bfa7625b698ec6e09dad7e1e2195d6fcda2c12c8631" Nov 04 01:15:49.241003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.234244 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:49.241003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.235635 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" event={"ID":"cf97e290-05ba-490e-834c-d34e195b76b3","Type":"ContainerStarted","Data":"c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f"} Nov 04 01:15:49.497770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.497338 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:15:49.497770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.497394 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 04 01:15:49.669008 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:49.665526 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf70af16_418f_4fa5_a904_75988861ce47.slice/crio-de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444 WatchSource:0}: Error finding container de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444: Status 404 returned error can't find the container with id de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444 Nov 04 01:15:49.670818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:49.669602 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs"] Nov 04 01:15:50.067439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.067415 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:50.069422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.068525 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:50.070136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.070095 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:15:50.261513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.261411 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" event={"ID":"cf97e290-05ba-490e-834c-d34e195b76b3","Type":"ContainerStarted","Data":"4824b198356921b9ea6e0d8d0a53aff14efd66eeaabed2bad4b89642aa5eb108"} Nov 04 01:15:50.264084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.264044 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" event={"ID":"bf70af16-418f-4fa5-a904-75988861ce47","Type":"ContainerStarted","Data":"215c28d55da41ea85c00252dd9b2f5963292606e1a09741a0bef99d782291d61"} Nov 04 01:15:50.264225 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.264106 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" event={"ID":"bf70af16-418f-4fa5-a904-75988861ce47","Type":"ContainerStarted","Data":"de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444"} Nov 04 01:15:50.418015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.417942 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" podStartSLOduration=2.417922697 podStartE2EDuration="2.417922697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:50.415147447 +0000 UTC m=+1119.173809609" watchObservedRunningTime="2026-11-04 01:15:50.417922697 +0000 UTC m=+1119.176584867" Nov 04 01:15:50.496936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.496906 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:15:50.497132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.496953 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 04 01:15:50.523571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.523488 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" podStartSLOduration=2.5234747779999998 podStartE2EDuration="2.523474778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:50.522058598 +0000 UTC m=+1119.280720759" watchObservedRunningTime="2026-11-04 01:15:50.523474778 +0000 UTC m=+1119.282136939" Nov 04 01:15:50.761502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.761469 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2"] Nov 04 01:15:50.765905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.765887 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.801377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.801352 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2"] Nov 04 01:15:50.851008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.828379 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.851008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.828528 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxt8\" (UniqueName: \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.851008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.828614 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.936009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.933605 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.936009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.933701 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.936009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.933812 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxt8\" (UniqueName: \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.936009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.934839 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.959696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.959666 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:50.988029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:50.988005 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxt8\" (UniqueName: \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\") pod \"maas-api-revoke-sub-044e8eab-r2mh2\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:51.113569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.113535 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:51.275998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.275950 4486 generic.go:356] "Generic (PLEG): container finished" podID="bf70af16-418f-4fa5-a904-75988861ce47" containerID="215c28d55da41ea85c00252dd9b2f5963292606e1a09741a0bef99d782291d61" exitCode=0 Nov 04 01:15:51.276175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.276078 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" event={"ID":"bf70af16-418f-4fa5-a904-75988861ce47","Type":"ContainerDied","Data":"215c28d55da41ea85c00252dd9b2f5963292606e1a09741a0bef99d782291d61"} Nov 04 01:15:51.278061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.278025 4486 generic.go:356] "Generic (PLEG): container finished" podID="cf97e290-05ba-490e-834c-d34e195b76b3" containerID="4824b198356921b9ea6e0d8d0a53aff14efd66eeaabed2bad4b89642aa5eb108" exitCode=0 Nov 04 01:15:51.278191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.278088 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" event={"ID":"cf97e290-05ba-490e-834c-d34e195b76b3","Type":"ContainerDied","Data":"4824b198356921b9ea6e0d8d0a53aff14efd66eeaabed2bad4b89642aa5eb108"} Nov 04 01:15:51.496805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.496721 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:15:51.496805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.496774 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 04 01:15:51.544014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:51.542832 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2"] Nov 04 01:15:52.287254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.287217 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" event={"ID":"864b6246-557c-4e6d-8948-b8227df04e67","Type":"ContainerStarted","Data":"a80512a36077f5b446ef4f990e5493f51bb24e6e113d61347d982f48f898ab83"} Nov 04 01:15:52.287745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.287281 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" event={"ID":"864b6246-557c-4e6d-8948-b8227df04e67","Type":"ContainerStarted","Data":"1a759a681c36d8f9176c8ee4a84dfa3ee550d67ba6d0eb80bf7dd9040baab10f"} Nov 04 01:15:52.404146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.404095 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" podStartSLOduration=2.4040793369999998 podStartE2EDuration="2.404079337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:52.401557606 +0000 UTC m=+1121.160219766" watchObservedRunningTime="2026-11-04 01:15:52.404079337 +0000 UTC m=+1121.162741498" Nov 04 01:15:52.496895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.496861 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:15:52.497077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.496915 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 04 01:15:52.510515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.509518 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:52.575221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.575180 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:52.586177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.586153 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\") pod \"cf97e290-05ba-490e-834c-d34e195b76b3\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " Nov 04 01:15:52.586322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.586260 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\") pod \"cf97e290-05ba-490e-834c-d34e195b76b3\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " Nov 04 01:15:52.586370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.586321 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\" (UniqueName: \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\") pod \"cf97e290-05ba-490e-834c-d34e195b76b3\" (UID: \"cf97e290-05ba-490e-834c-d34e195b76b3\") " Nov 04 01:15:52.586502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.586473 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca" pod "cf97e290-05ba-490e-834c-d34e195b76b3" (UID: "cf97e290-05ba-490e-834c-d34e195b76b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:52.586800 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.586777 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:52.588413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.588389 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca" pod "cf97e290-05ba-490e-834c-d34e195b76b3" (UID: "cf97e290-05ba-490e-834c-d34e195b76b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:52.588501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.588476 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r" pod "cf97e290-05ba-490e-834c-d34e195b76b3" (UID: "cf97e290-05ba-490e-834c-d34e195b76b3"). InnerVolumeSpecName "kube-api-access-ghd2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:52.688079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.687606 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\") pod \"bf70af16-418f-4fa5-a904-75988861ce47\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " Nov 04 01:15:52.688267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.688090 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\" (UniqueName: \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\") pod \"bf70af16-418f-4fa5-a904-75988861ce47\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " Nov 04 01:15:52.689035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.688342 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\") pod \"bf70af16-418f-4fa5-a904-75988861ce47\" (UID: \"bf70af16-418f-4fa5-a904-75988861ce47\") " Nov 04 01:15:52.689035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.688746 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca" pod "bf70af16-418f-4fa5-a904-75988861ce47" (UID: "bf70af16-418f-4fa5-a904-75988861ce47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:52.689035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.689013 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:52.689035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.689037 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghd2r\" (UniqueName: \"kubernetes.io/projected/cf97e290-05ba-490e-834c-d34e195b76b3-kube-api-access-ghd2r\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:52.689316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.689051 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf97e290-05ba-490e-834c-d34e195b76b3-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:52.692326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.691908 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht" pod "bf70af16-418f-4fa5-a904-75988861ce47" (UID: "bf70af16-418f-4fa5-a904-75988861ce47"). InnerVolumeSpecName "kube-api-access-qdfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:52.695031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.694935 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca" pod "bf70af16-418f-4fa5-a904-75988861ce47" (UID: "bf70af16-418f-4fa5-a904-75988861ce47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:52.790438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.790403 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf70af16-418f-4fa5-a904-75988861ce47-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:52.790438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:52.790436 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfht\" (UniqueName: \"kubernetes.io/projected/bf70af16-418f-4fa5-a904-75988861ce47-kube-api-access-qdfht\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:53.297304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.297217 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" Nov 04 01:15:53.297304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.297258 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97" event={"ID":"cf97e290-05ba-490e-834c-d34e195b76b3","Type":"ContainerDied","Data":"c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f"} Nov 04 01:15:53.297749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.297318 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51f0fa7da642b4217e5aaada34786965b917b9c7799f0b8cc1ce796413c156f" Nov 04 01:15:53.298960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.298938 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" Nov 04 01:15:53.299117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.298932 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs" event={"ID":"bf70af16-418f-4fa5-a904-75988861ce47","Type":"ContainerDied","Data":"de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444"} Nov 04 01:15:53.299117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.299030 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de91920170d7081f81dbe7fc6312bc324154cac0c07ffd8a1322554b42ae4444" Nov 04 01:15:53.301170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.301145 4486 generic.go:356] "Generic (PLEG): container finished" podID="864b6246-557c-4e6d-8948-b8227df04e67" containerID="a80512a36077f5b446ef4f990e5493f51bb24e6e113d61347d982f48f898ab83" exitCode=0 Nov 04 01:15:53.301281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.301187 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" event={"ID":"864b6246-557c-4e6d-8948-b8227df04e67","Type":"ContainerDied","Data":"a80512a36077f5b446ef4f990e5493f51bb24e6e113d61347d982f48f898ab83"} Nov 04 01:15:53.744150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.744114 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:53.744999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.744964 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:53.746283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:53.746252 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:15:54.471858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.471826 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:54.510715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.510690 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\") pod \"864b6246-557c-4e6d-8948-b8227df04e67\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " Nov 04 01:15:54.510859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.510746 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\" (UniqueName: \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\") pod \"864b6246-557c-4e6d-8948-b8227df04e67\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " Nov 04 01:15:54.510859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.510819 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\") pod \"864b6246-557c-4e6d-8948-b8227df04e67\" (UID: \"864b6246-557c-4e6d-8948-b8227df04e67\") " Nov 04 01:15:54.511180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.511145 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca" pod "864b6246-557c-4e6d-8948-b8227df04e67" (UID: "864b6246-557c-4e6d-8948-b8227df04e67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:15:54.511283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.511269 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:54.512776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.512756 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca" pod "864b6246-557c-4e6d-8948-b8227df04e67" (UID: "864b6246-557c-4e6d-8948-b8227df04e67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:54.512887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.512866 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8" pod "864b6246-557c-4e6d-8948-b8227df04e67" (UID: "864b6246-557c-4e6d-8948-b8227df04e67"). InnerVolumeSpecName "kube-api-access-csxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:54.612229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.612194 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864b6246-557c-4e6d-8948-b8227df04e67-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:54.612229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:54.612223 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csxt8\" (UniqueName: \"kubernetes.io/projected/864b6246-557c-4e6d-8948-b8227df04e67-kube-api-access-csxt8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:55.275330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275296 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:15:55.275853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275840 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bf70af16-418f-4fa5-a904-75988861ce47" containerName="revoke-keys" Nov 04 01:15:55.275908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275853 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf70af16-418f-4fa5-a904-75988861ce47" containerName="revoke-keys" Nov 04 01:15:55.275908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275865 4486 cpu_manager.go:414] "containerMap: removing container" podUID="cf97e290-05ba-490e-834c-d34e195b76b3" containerName="revoke-keys" Nov 04 01:15:55.275908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275870 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf97e290-05ba-490e-834c-d34e195b76b3" containerName="revoke-keys" Nov 04 01:15:55.275908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275896 4486 cpu_manager.go:414] "containerMap: removing container" podUID="864b6246-557c-4e6d-8948-b8227df04e67" containerName="revoke-keys" Nov 04 01:15:55.275908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275901 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864b6246-557c-4e6d-8948-b8227df04e67" containerName="revoke-keys" Nov 04 01:15:55.276098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.275992 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="864b6246-557c-4e6d-8948-b8227df04e67" containerName="revoke-keys" Nov 04 01:15:55.276098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.276005 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf70af16-418f-4fa5-a904-75988861ce47" containerName="revoke-keys" Nov 04 01:15:55.276098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.276012 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf97e290-05ba-490e-834c-d34e195b76b3" containerName="revoke-keys" Nov 04 01:15:55.279580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.279561 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.316648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.316619 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" Nov 04 01:15:55.316768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.316643 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2" event={"ID":"864b6246-557c-4e6d-8948-b8227df04e67","Type":"ContainerDied","Data":"1a759a681c36d8f9176c8ee4a84dfa3ee550d67ba6d0eb80bf7dd9040baab10f"} Nov 04 01:15:55.316768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.316689 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a759a681c36d8f9176c8ee4a84dfa3ee550d67ba6d0eb80bf7dd9040baab10f" Nov 04 01:15:55.319706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.319679 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.319831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.319718 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.319831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.319749 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.319962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.319902 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.320056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.319961 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.320121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.320055 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.320121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.320087 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrjs\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.320238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.320170 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.320293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.320235 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.380785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.380752 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:15:55.422014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.421948 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422029 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422086 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422120 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422214 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422283 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422317 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrjs\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422529 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422603 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422732 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.422827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.422800 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.423055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.423031 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.423155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.423123 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.423624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.423602 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.424589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.424563 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.424646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.424626 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.476171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.476147 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.476467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.476328 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrjs\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\") pod \"maas-default-gateway-openshift-default-694db55c87-9tzv9\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:55.598555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:55.598523 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:56.019306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.019274 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:15:56.019744 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:15:56.019712 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd91c68_fd7d_4b3c_95aa_55680be62f4d.slice/crio-d5082362b197c656ab8a803455def588458f1b189cb598648c47579dd4183003 WatchSource:0}: Error finding container d5082362b197c656ab8a803455def588458f1b189cb598648c47579dd4183003: Status 404 returned error can't find the container with id d5082362b197c656ab8a803455def588458f1b189cb598648c47579dd4183003 Nov 04 01:15:56.022511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.022471 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:56.022579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.022559 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:56.022626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.022611 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:15:56.362003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.361172 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" event={"ID":"9fd91c68-fd7d-4b3c-95aa-55680be62f4d","Type":"ContainerStarted","Data":"daf9dafb7a24160376dd14f057004830e76e06c4beb1a814af45e312ace2dda1"} Nov 04 01:15:56.362003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.361239 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" event={"ID":"9fd91c68-fd7d-4b3c-95aa-55680be62f4d","Type":"ContainerStarted","Data":"d5082362b197c656ab8a803455def588458f1b189cb598648c47579dd4183003"} Nov 04 01:15:56.505005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.489268 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podStartSLOduration=1.489244719 podStartE2EDuration="1.489244719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:56.474450788 +0000 UTC m=+1125.233112974" watchObservedRunningTime="2026-11-04 01:15:56.489244719 +0000 UTC m=+1125.247906883" Nov 04 01:15:56.620997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:56.620081 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:15:57.600875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:57.600830 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:15:57.601334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:57.600903 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:15:58.599756 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:58.599715 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:15:58.599920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:58.599782 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:15:59.599476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.599442 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:15:59.600020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.599496 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:15:59.658493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.658444 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb"] Nov 04 01:15:59.673369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.673341 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.703493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.702783 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb"] Nov 04 01:15:59.754006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.753256 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:15:59.754006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.753362 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:15:59.777026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.760674 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.777026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.760842 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kxs\" (UniqueName: \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.777026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.760907 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.865726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.865630 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.865878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.865757 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.865928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.865899 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kxs\" (UniqueName: \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.866883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.866861 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.868887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.868859 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.919842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.919815 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kxs\" (UniqueName: \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\") pod \"maas-api-revoke-sub-4202f8d0-tgglb\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:15:59.993998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:15:59.993948 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:16:00.400221 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:00.400179 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571fee_a1bd_414d_8365_ab602361e074.slice/crio-8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add WatchSource:0}: Error finding container 8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add: Status 404 returned error can't find the container with id 8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add Nov 04 01:16:00.400221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:00.400202 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb"] Nov 04 01:16:00.473430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:00.473396 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" event={"ID":"a5571fee-a1bd-414d-8365-ab602361e074","Type":"ContainerStarted","Data":"8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add"} Nov 04 01:16:00.599638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:00.599599 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:16:00.600101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:00.599654 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:16:01.482532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.482489 4486 generic.go:356] "Generic (PLEG): container finished" podID="a5571fee-a1bd-414d-8365-ab602361e074" containerID="fcd61f61f0408f7c8357bfb30ba0d5f0374f4624f7e75dcc4856c4048826039e" exitCode=0 Nov 04 01:16:01.482698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.482580 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" event={"ID":"a5571fee-a1bd-414d-8365-ab602361e074","Type":"ContainerDied","Data":"fcd61f61f0408f7c8357bfb30ba0d5f0374f4624f7e75dcc4856c4048826039e"} Nov 04 01:16:01.585637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.585597 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5"] Nov 04 01:16:01.589763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.589739 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.599867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.599825 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:16:01.600279 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.599888 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:16:01.617470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.617433 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5"] Nov 04 01:16:01.688679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.688565 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvsk\" (UniqueName: \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.688679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.688670 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.688930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.688829 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.790378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.790332 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.790558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.790478 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvsk\" (UniqueName: \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.790558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.790536 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.791608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.791578 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.798476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.797462 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.867086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.867049 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvsk\" (UniqueName: \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\") pod \"maas-api-revoke-sub-c0cd2e39-64rs5\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:01.904071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:01.904025 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:02.313924 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:02.313787 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e7f85d_f9e0_4f3d_9bb1_0070d14d24b3.slice/crio-6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b WatchSource:0}: Error finding container 6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b: Status 404 returned error can't find the container with id 6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b Nov 04 01:16:02.313924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.313806 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5"] Nov 04 01:16:02.466658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.466618 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv"] Nov 04 01:16:02.472353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.472323 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.491448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.491406 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" event={"ID":"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3","Type":"ContainerStarted","Data":"6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b"} Nov 04 01:16:02.518903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.518863 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv"] Nov 04 01:16:02.600283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.600243 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:16:02.600787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.600315 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:16:02.600787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.600753 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.600944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.600865 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.600944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.600908 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldpg\" (UniqueName: \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.701760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.701709 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.701941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.701806 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldpg\" (UniqueName: \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.701941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.701891 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.702675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.702647 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.706760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.706728 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.730626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.730595 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:16:02.757568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.757530 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldpg\" (UniqueName: \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\") pod \"maas-api-revoke-sub-128f4abf-v57nv\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.787213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.787179 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:02.904860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.904818 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\") pod \"a5571fee-a1bd-414d-8365-ab602361e074\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " Nov 04 01:16:02.905102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.905007 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\") pod \"a5571fee-a1bd-414d-8365-ab602361e074\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " Nov 04 01:16:02.905102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.905073 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\" (UniqueName: \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\") pod \"a5571fee-a1bd-414d-8365-ab602361e074\" (UID: \"a5571fee-a1bd-414d-8365-ab602361e074\") " Nov 04 01:16:02.906540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.906510 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca" pod "a5571fee-a1bd-414d-8365-ab602361e074" (UID: "a5571fee-a1bd-414d-8365-ab602361e074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:02.909566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.909429 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca" pod "a5571fee-a1bd-414d-8365-ab602361e074" (UID: "a5571fee-a1bd-414d-8365-ab602361e074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:02.911546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:02.911507 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs" pod "a5571fee-a1bd-414d-8365-ab602361e074" (UID: "a5571fee-a1bd-414d-8365-ab602361e074"). InnerVolumeSpecName "kube-api-access-j5kxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:03.007544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.007496 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:03.007544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.007549 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5kxs\" (UniqueName: \"kubernetes.io/projected/a5571fee-a1bd-414d-8365-ab602361e074-kube-api-access-j5kxs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:03.007749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.007567 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5571fee-a1bd-414d-8365-ab602361e074-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:03.185591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.185555 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv"] Nov 04 01:16:03.186925 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:03.186896 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4180686_7c78_4fbf_98cc_34a5507cf425.slice/crio-672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605 WatchSource:0}: Error finding container 672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605: Status 404 returned error can't find the container with id 672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605 Nov 04 01:16:03.500045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.500008 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" event={"ID":"d4180686-7c78-4fbf-98cc-34a5507cf425","Type":"ContainerStarted","Data":"672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605"} Nov 04 01:16:03.501541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.501513 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" Nov 04 01:16:03.501702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.501506 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb" event={"ID":"a5571fee-a1bd-414d-8365-ab602361e074","Type":"ContainerDied","Data":"8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add"} Nov 04 01:16:03.501702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.501638 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8beb4dec08e4036746e4277894e262da0be95fa5b92daaa82bbe9fa4d9fc8add" Nov 04 01:16:03.503265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.503238 4486 generic.go:356] "Generic (PLEG): container finished" podID="59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" containerID="1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605" exitCode=0 Nov 04 01:16:03.503365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.503319 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" event={"ID":"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3","Type":"ContainerDied","Data":"1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605"} Nov 04 01:16:03.599385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.599348 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 04 01:16:03.599584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.599409 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 04 01:16:03.649739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.649700 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:16:03.650515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.650491 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a5571fee-a1bd-414d-8365-ab602361e074" containerName="revoke-keys" Nov 04 01:16:03.650610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.650517 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5571fee-a1bd-414d-8365-ab602361e074" containerName="revoke-keys" Nov 04 01:16:03.650715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.650700 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5571fee-a1bd-414d-8365-ab602361e074" containerName="revoke-keys" Nov 04 01:16:03.654286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.654261 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.704440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.704313 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"distinct-w2-2fd27d-kserve-self-signed-certs\"" Nov 04 01:16:03.704440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.704336 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"openshift-service-ca.crt\"" Nov 04 01:16:03.704440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.704389 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"kube-root-ca.crt\"" Nov 04 01:16:03.704440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.704418 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2fd27d\"/\"default-dockercfg-kvnqv\"" Nov 04 01:16:03.722732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.722687 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:16:03.818828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.818783 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfrmn\" (UniqueName: \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.819023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.818898 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.819023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.819010 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.819166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.819044 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.819166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.819097 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.819166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.819145 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.919973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.919927 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920001 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920069 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920124 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920250 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfrmn\" (UniqueName: \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920343 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920502 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920532 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.920828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.920701 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.923196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.923174 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.928648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.928618 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:03.975091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:03.974997 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfrmn\" (UniqueName: \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\") pod \"distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:04.268274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.268175 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:04.515281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.515246 4486 generic.go:356] "Generic (PLEG): container finished" podID="d4180686-7c78-4fbf-98cc-34a5507cf425" containerID="018990968fb0353a52af27b6aa738dea7a87d757b8fc43841e3d48503064be78" exitCode=0 Nov 04 01:16:04.515443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.515339 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" event={"ID":"d4180686-7c78-4fbf-98cc-34a5507cf425","Type":"ContainerDied","Data":"018990968fb0353a52af27b6aa738dea7a87d757b8fc43841e3d48503064be78"} Nov 04 01:16:04.675301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.675274 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:04.688067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.687306 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:16:04.845718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.844694 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\") pod \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " Nov 04 01:16:04.845718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.844771 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\" (UniqueName: \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\") pod \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " Nov 04 01:16:04.845718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.844898 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\") pod \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\" (UID: \"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3\") " Nov 04 01:16:04.846391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.846322 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca" pod "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" (UID: "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:04.847949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.847922 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca" pod "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" (UID: "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:04.848434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.848410 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk" pod "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" (UID: "59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3"). InnerVolumeSpecName "kube-api-access-8rvsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:04.946470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.946430 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvsk\" (UniqueName: \"kubernetes.io/projected/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-kube-api-access-8rvsk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:04.946470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.946463 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:04.946648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:04.946478 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:05.525293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.525186 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" event={"ID":"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2","Type":"ContainerStarted","Data":"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599"} Nov 04 01:16:05.525293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.525245 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" event={"ID":"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2","Type":"ContainerStarted","Data":"2c2d5de3ca78172f61f15f23406f46a63831ec31a829a8aecaca4d5cdbf6081c"} Nov 04 01:16:05.526691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.526671 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" Nov 04 01:16:05.526797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.526708 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5" event={"ID":"59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3","Type":"ContainerDied","Data":"6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b"} Nov 04 01:16:05.526797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.526751 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c615e741f2bdb6c3d7b2c3a670622a76f207ce7b8bacf2c69bf0ff42e32ee8b" Nov 04 01:16:05.603535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.603486 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:05.603698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.603566 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:05.636755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.636704 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podStartSLOduration=2.636690759 podStartE2EDuration="2.636690759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:05.633688105 +0000 UTC m=+1134.392350296" watchObservedRunningTime="2026-11-04 01:16:05.636690759 +0000 UTC m=+1134.395352920" Nov 04 01:16:05.674097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.674069 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:05.865028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.864998 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\" (UniqueName: \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\") pod \"d4180686-7c78-4fbf-98cc-34a5507cf425\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " Nov 04 01:16:05.865524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.865123 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\") pod \"d4180686-7c78-4fbf-98cc-34a5507cf425\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " Nov 04 01:16:05.865524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.865218 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\") pod \"d4180686-7c78-4fbf-98cc-34a5507cf425\" (UID: \"d4180686-7c78-4fbf-98cc-34a5507cf425\") " Nov 04 01:16:05.865618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.865544 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca" pod "d4180686-7c78-4fbf-98cc-34a5507cf425" (UID: "d4180686-7c78-4fbf-98cc-34a5507cf425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:05.865766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.865750 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:05.867057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.867031 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg" pod "d4180686-7c78-4fbf-98cc-34a5507cf425" (UID: "d4180686-7c78-4fbf-98cc-34a5507cf425"). InnerVolumeSpecName "kube-api-access-8ldpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:05.867147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.867114 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca" pod "d4180686-7c78-4fbf-98cc-34a5507cf425" (UID: "d4180686-7c78-4fbf-98cc-34a5507cf425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:05.966447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.966418 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldpg\" (UniqueName: \"kubernetes.io/projected/d4180686-7c78-4fbf-98cc-34a5507cf425-kube-api-access-8ldpg\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:05.966447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:05.966447 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4180686-7c78-4fbf-98cc-34a5507cf425-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:06.271151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.271067 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:06.536597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.536498 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" event={"ID":"d4180686-7c78-4fbf-98cc-34a5507cf425","Type":"ContainerDied","Data":"672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605"} Nov 04 01:16:06.536597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.536560 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="672561d7a0c9637134f272c5ede9f15252ae9a282d300d37226f3c32ffe1f605" Nov 04 01:16:06.536597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.536516 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv" Nov 04 01:16:06.648503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.648466 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-9tzv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:06.648767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:06.648736 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:07.566844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:07.566814 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:16:07.567390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:07.567237 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:16:07.574462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:07.568322 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:16:12.787779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.787744 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx"] Nov 04 01:16:12.788406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788388 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d4180686-7c78-4fbf-98cc-34a5507cf425" containerName="revoke-keys" Nov 04 01:16:12.788445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788407 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4180686-7c78-4fbf-98cc-34a5507cf425" containerName="revoke-keys" Nov 04 01:16:12.788445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788422 4486 cpu_manager.go:414] "containerMap: removing container" podUID="59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" containerName="revoke-keys" Nov 04 01:16:12.788445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788427 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" containerName="revoke-keys" Nov 04 01:16:12.788549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788543 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4180686-7c78-4fbf-98cc-34a5507cf425" containerName="revoke-keys" Nov 04 01:16:12.788581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.788550 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" containerName="revoke-keys" Nov 04 01:16:12.793675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.793649 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.844795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.844772 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx"] Nov 04 01:16:12.849496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.849464 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8hn\" (UniqueName: \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.849612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.849529 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.849677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.849654 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.886796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.886770 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:16:12.886888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.886853 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:16:12.950549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.950519 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.950682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.950646 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8hn\" (UniqueName: \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.950682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.950680 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.951224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.951201 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:12.953022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:12.953001 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:13.010612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:13.010584 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8hn\" (UniqueName: \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\") pod \"maas-api-revoke-sub-062c8bcb-qs4sx\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:13.107617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:13.107579 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:13.480433 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:13.480398 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21a68e4_9d4d_4e9c_acde_eaf1c06b88eb.slice/crio-1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d WatchSource:0}: Error finding container 1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d: Status 404 returned error can't find the container with id 1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d Nov 04 01:16:13.481073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:13.481047 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx"] Nov 04 01:16:13.597525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:13.597489 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" event={"ID":"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb","Type":"ContainerStarted","Data":"1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d"} Nov 04 01:16:13.830311 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:16:13.830279 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:16:14.269268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.269241 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:14.285006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.283414 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8"] Nov 04 01:16:14.289973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.289943 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.290667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.290628 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:14.339597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.339563 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8"] Nov 04 01:16:14.381241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.381207 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9kl\" (UniqueName: \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.381374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.381251 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.381414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.381367 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.482995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.482954 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9kl\" (UniqueName: \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.483138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.483040 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.483208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.483135 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.483772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.483743 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.485306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.485288 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.539208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.539136 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9kl\" (UniqueName: \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\") pod \"maas-api-revoke-sub-dc5c79d4-mbng8\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.604708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.604670 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:14.606427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.606401 4486 generic.go:356] "Generic (PLEG): container finished" podID="d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" containerID="e0bb871842c2fd20214bb0d080494a7f6d48f7204ac95116a8235746f2621eb4" exitCode=0 Nov 04 01:16:14.606548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.606483 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" event={"ID":"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb","Type":"ContainerDied","Data":"e0bb871842c2fd20214bb0d080494a7f6d48f7204ac95116a8235746f2621eb4"} Nov 04 01:16:14.618140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.618120 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:16:14.947124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.947088 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr"] Nov 04 01:16:14.953066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.953039 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:14.987696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:14.987662 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8"] Nov 04 01:16:14.988375 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:14.988325 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9251a9d4_b817_40b4_ac4f_8ad4e3e6ec29.slice/crio-731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083 WatchSource:0}: Error finding container 731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083: Status 404 returned error can't find the container with id 731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083 Nov 04 01:16:15.004462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.004427 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr"] Nov 04 01:16:15.092434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.092390 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfj9\" (UniqueName: \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.092597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.092565 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.092637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.092614 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.194124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.194067 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.194304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.194161 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.194304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.194265 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfj9\" (UniqueName: \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.194746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.194720 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.196611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.196585 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.248613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.248531 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfj9\" (UniqueName: \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\") pod \"maas-api-revoke-sub-49fd4126-gwfjr\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.280839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.280803 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:15.617396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.617349 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" event={"ID":"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29","Type":"ContainerStarted","Data":"52438940aed9c9ace7258a71b3bed1d5db66bd950f1bc17063c2064ee7004705"} Nov 04 01:16:15.617578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.617418 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" event={"ID":"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29","Type":"ContainerStarted","Data":"731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083"} Nov 04 01:16:15.659885 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:15.659855 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd252a4f3_c1aa_4dec_884f_7a9d46ffa849.slice/crio-cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa WatchSource:0}: Error finding container cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa: Status 404 returned error can't find the container with id cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa Nov 04 01:16:15.666577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.666545 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr"] Nov 04 01:16:15.730919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.730851 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" podStartSLOduration=1.730829697 podStartE2EDuration="1.730829697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:15.725361722 +0000 UTC m=+1144.484023884" watchObservedRunningTime="2026-11-04 01:16:15.730829697 +0000 UTC m=+1144.489491866" Nov 04 01:16:15.852364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:15.852334 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:16.003387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.003346 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\" (UniqueName: \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\") pod \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " Nov 04 01:16:16.003919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.003451 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\") pod \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " Nov 04 01:16:16.003919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.003637 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\") pod \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\" (UID: \"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb\") " Nov 04 01:16:16.004251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.004200 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca" pod "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" (UID: "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:16.006185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.006155 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca" pod "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" (UID: "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:16.006374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.006357 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn" pod "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" (UID: "d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb"). InnerVolumeSpecName "kube-api-access-gd8hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:16.105355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.105319 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:16.105355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.105353 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8hn\" (UniqueName: \"kubernetes.io/projected/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-kube-api-access-gd8hn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:16.105591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.105369 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:16.633013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.632949 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" event={"ID":"d252a4f3-c1aa-4dec-884f-7a9d46ffa849","Type":"ContainerStarted","Data":"5b74dea08064480f2a887ab41f90fc96042bc8c801dd2f14906c91efe8e0ecd2"} Nov 04 01:16:16.633167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.633030 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" event={"ID":"d252a4f3-c1aa-4dec-884f-7a9d46ffa849","Type":"ContainerStarted","Data":"cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa"} Nov 04 01:16:16.634917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.634886 4486 generic.go:356] "Generic (PLEG): container finished" podID="9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" containerID="52438940aed9c9ace7258a71b3bed1d5db66bd950f1bc17063c2064ee7004705" exitCode=0 Nov 04 01:16:16.635088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.635036 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" event={"ID":"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29","Type":"ContainerDied","Data":"52438940aed9c9ace7258a71b3bed1d5db66bd950f1bc17063c2064ee7004705"} Nov 04 01:16:16.636645 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.636615 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" Nov 04 01:16:16.636754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.636661 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx" event={"ID":"d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb","Type":"ContainerDied","Data":"1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d"} Nov 04 01:16:16.636754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.636711 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1861cb809a3fd55a5b1155793cb1504e24def2e4c4c4d955192b4addbd287a3d" Nov 04 01:16:16.744626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:16.744569 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" podStartSLOduration=2.7445531 podStartE2EDuration="2.7445531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:16.741024999 +0000 UTC m=+1145.499687160" watchObservedRunningTime="2026-11-04 01:16:16.7445531 +0000 UTC m=+1145.503215260" Nov 04 01:16:17.647877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.647835 4486 generic.go:356] "Generic (PLEG): container finished" podID="d252a4f3-c1aa-4dec-884f-7a9d46ffa849" containerID="5b74dea08064480f2a887ab41f90fc96042bc8c801dd2f14906c91efe8e0ecd2" exitCode=0 Nov 04 01:16:17.648404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.647930 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" event={"ID":"d252a4f3-c1aa-4dec-884f-7a9d46ffa849","Type":"ContainerDied","Data":"5b74dea08064480f2a887ab41f90fc96042bc8c801dd2f14906c91efe8e0ecd2"} Nov 04 01:16:17.821861 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.821829 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:17.927914 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.927824 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\") pod \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " Nov 04 01:16:17.927914 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.927905 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\") pod \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " Nov 04 01:16:17.928162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.928131 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\" (UniqueName: \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\") pod \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\" (UID: \"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29\") " Nov 04 01:16:17.928377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.928344 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca" pod "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" (UID: "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:17.928591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.928572 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:17.944156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.930269 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca" pod "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" (UID: "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:17.944156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:17.930709 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl" pod "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" (UID: "9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29"). InnerVolumeSpecName "kube-api-access-nw9kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:18.029652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.029611 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9kl\" (UniqueName: \"kubernetes.io/projected/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-kube-api-access-nw9kl\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:18.029652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.029654 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:18.658431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.658392 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" Nov 04 01:16:18.658829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.658424 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8" event={"ID":"9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29","Type":"ContainerDied","Data":"731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083"} Nov 04 01:16:18.658829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.658485 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="731624c6179627c583952b2da5b2bafb6def3edd6dcc71123cdd3995687c6083" Nov 04 01:16:18.816446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.816415 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:18.939727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.939628 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\") pod \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " Nov 04 01:16:18.939727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.939696 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\") pod \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " Nov 04 01:16:18.939965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.939841 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\" (UniqueName: \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\") pod \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\" (UID: \"d252a4f3-c1aa-4dec-884f-7a9d46ffa849\") " Nov 04 01:16:18.940168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.940136 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca" pod "d252a4f3-c1aa-4dec-884f-7a9d46ffa849" (UID: "d252a4f3-c1aa-4dec-884f-7a9d46ffa849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:18.941030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.940954 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:18.942066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.942042 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca" pod "d252a4f3-c1aa-4dec-884f-7a9d46ffa849" (UID: "d252a4f3-c1aa-4dec-884f-7a9d46ffa849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:18.942509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:18.942492 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9" pod "d252a4f3-c1aa-4dec-884f-7a9d46ffa849" (UID: "d252a4f3-c1aa-4dec-884f-7a9d46ffa849"). InnerVolumeSpecName "kube-api-access-kkfj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:19.042116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:19.042069 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:19.042116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:19.042105 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfj9\" (UniqueName: \"kubernetes.io/projected/d252a4f3-c1aa-4dec-884f-7a9d46ffa849-kube-api-access-kkfj9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:19.669690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:19.669590 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" Nov 04 01:16:19.669690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:19.669612 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr" event={"ID":"d252a4f3-c1aa-4dec-884f-7a9d46ffa849","Type":"ContainerDied","Data":"cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa"} Nov 04 01:16:19.669690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:19.669674 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba519ab2432159975755a4fad9d674bd33f57f7d366099c8dc4661341bf05fa" Nov 04 01:16:29.796167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.796123 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx"] Nov 04 01:16:29.796967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.796943 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" containerName="revoke-keys" Nov 04 01:16:29.797112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.796970 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" containerName="revoke-keys" Nov 04 01:16:29.797112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797036 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d252a4f3-c1aa-4dec-884f-7a9d46ffa849" containerName="revoke-keys" Nov 04 01:16:29.797112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797047 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d252a4f3-c1aa-4dec-884f-7a9d46ffa849" containerName="revoke-keys" Nov 04 01:16:29.797112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797084 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" containerName="revoke-keys" Nov 04 01:16:29.797112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797092 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" containerName="revoke-keys" Nov 04 01:16:29.797357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797237 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" containerName="revoke-keys" Nov 04 01:16:29.797357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797256 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" containerName="revoke-keys" Nov 04 01:16:29.797357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.797271 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d252a4f3-c1aa-4dec-884f-7a9d46ffa849" containerName="revoke-keys" Nov 04 01:16:29.800679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.800655 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.832167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.832126 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.832352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.832191 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjgz\" (UniqueName: \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.832395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.832347 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.856778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.856739 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx"] Nov 04 01:16:29.897136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.897095 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:16:29.897325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.897151 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:16:29.933212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.933173 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.933426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.933326 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.933426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.933370 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjgz\" (UniqueName: \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.933846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.933825 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.935675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.935649 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:29.987992 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:29.987950 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjgz\" (UniqueName: \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\") pod \"maas-api-revoke-sub-45bddcc9-nptnx\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:30.114344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:30.114302 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:30.490735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:30.490276 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx"] Nov 04 01:16:30.491046 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:30.490977 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899d8915_3cf4_4bb3_bb2f_4aee0920417e.slice/crio-21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f WatchSource:0}: Error finding container 21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f: Status 404 returned error can't find the container with id 21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f Nov 04 01:16:30.771360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:30.771326 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" event={"ID":"899d8915-3cf4-4bb3-bb2f-4aee0920417e","Type":"ContainerStarted","Data":"21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f"} Nov 04 01:16:31.781097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:31.780969 4486 generic.go:356] "Generic (PLEG): container finished" podID="899d8915-3cf4-4bb3-bb2f-4aee0920417e" containerID="b7d2cab5bbbd76a7595d08eaa88cf8ffa5e13a3c2316db61e3ab865a76b734bb" exitCode=0 Nov 04 01:16:31.781097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:31.781036 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" event={"ID":"899d8915-3cf4-4bb3-bb2f-4aee0920417e","Type":"ContainerDied","Data":"b7d2cab5bbbd76a7595d08eaa88cf8ffa5e13a3c2316db61e3ab865a76b734bb"} Nov 04 01:16:32.950626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:32.950597 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:33.071748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.071697 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\") pod \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " Nov 04 01:16:33.071952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.071785 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\") pod \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " Nov 04 01:16:33.071952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.071821 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\" (UniqueName: \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\") pod \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\" (UID: \"899d8915-3cf4-4bb3-bb2f-4aee0920417e\") " Nov 04 01:16:33.072357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.072324 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca" pod "899d8915-3cf4-4bb3-bb2f-4aee0920417e" (UID: "899d8915-3cf4-4bb3-bb2f-4aee0920417e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:16:33.074577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.074548 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz" pod "899d8915-3cf4-4bb3-bb2f-4aee0920417e" (UID: "899d8915-3cf4-4bb3-bb2f-4aee0920417e"). InnerVolumeSpecName "kube-api-access-zpjgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:16:33.074818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.074786 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca" pod "899d8915-3cf4-4bb3-bb2f-4aee0920417e" (UID: "899d8915-3cf4-4bb3-bb2f-4aee0920417e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:16:33.173700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.173659 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:33.173700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.173696 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/899d8915-3cf4-4bb3-bb2f-4aee0920417e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:33.173700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.173707 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjgz\" (UniqueName: \"kubernetes.io/projected/899d8915-3cf4-4bb3-bb2f-4aee0920417e-kube-api-access-zpjgz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:16:33.799464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.799375 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" Nov 04 01:16:33.799464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.799409 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx" event={"ID":"899d8915-3cf4-4bb3-bb2f-4aee0920417e","Type":"ContainerDied","Data":"21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f"} Nov 04 01:16:33.799652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:33.799467 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ad6d28aa9e48c224ff0f2ac9a9919d0339189940c728401035fda91e87164f" Nov 04 01:16:43.835447 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:16:43.835405 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:16:46.928435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.928403 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:16:46.929131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.929112 4486 cpu_manager.go:414] "containerMap: removing container" podUID="899d8915-3cf4-4bb3-bb2f-4aee0920417e" containerName="revoke-keys" Nov 04 01:16:46.929131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.929131 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899d8915-3cf4-4bb3-bb2f-4aee0920417e" containerName="revoke-keys" Nov 04 01:16:46.929260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.929232 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="899d8915-3cf4-4bb3-bb2f-4aee0920417e" containerName="revoke-keys" Nov 04 01:16:46.934203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.934186 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.953719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.953696 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.953858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.953820 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.953974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.953916 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.953974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.953959 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.954133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.954065 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.954133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.954089 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.954133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.954115 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.954274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.954179 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.954324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.954293 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztpx\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:46.990034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:46.990007 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:16:47.055069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055042 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055080 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055136 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztpx\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055167 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055221 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055253 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055287 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055342 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055587 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055601 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055682 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.055882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.055837 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.056236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.056216 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.057429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.057407 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.057775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.057748 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.109738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.109714 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.109855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.109821 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztpx\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.251161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.251082 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:47.686603 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:47.686574 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94101ccd_0f20_4e5c_8226_7adb50643c66.slice/crio-8d15608f4af2d2abb236ee8daba17dc2fa94f1b628915ce40ea83f521104fc51 WatchSource:0}: Error finding container 8d15608f4af2d2abb236ee8daba17dc2fa94f1b628915ce40ea83f521104fc51: Status 404 returned error can't find the container with id 8d15608f4af2d2abb236ee8daba17dc2fa94f1b628915ce40ea83f521104fc51 Nov 04 01:16:47.687601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.687571 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:16:47.688763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.688728 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:47.688856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.688809 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:47.688856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.688840 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:47.935659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.933433 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" event={"ID":"94101ccd-0f20-4e5c-8226-7adb50643c66","Type":"ContainerStarted","Data":"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b"} Nov 04 01:16:47.935659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.933522 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" event={"ID":"94101ccd-0f20-4e5c-8226-7adb50643c66","Type":"ContainerStarted","Data":"8d15608f4af2d2abb236ee8daba17dc2fa94f1b628915ce40ea83f521104fc51"} Nov 04 01:16:47.948021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.944077 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:16:47.949544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.949516 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965317 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965361 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbft\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965458 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965807 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965866 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.965921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965895 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.966085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.965960 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.966153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.966134 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:47.966209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:47.966169 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.004231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.004209 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:16:48.031172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.031153 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2ee045-openshift-default-dockercfg-2pv54\"" Nov 04 01:16:48.046765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.046722 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podStartSLOduration=2.046709963 podStartE2EDuration="2.046709963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:48.045939488 +0000 UTC m=+1176.804601652" watchObservedRunningTime="2026-11-04 01:16:48.046709963 +0000 UTC m=+1176.805372124" Nov 04 01:16:48.067019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.066961 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067022 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbft\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067064 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067130 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067158 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067174 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067237 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067313 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067341 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067387 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.067667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.067465 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.068076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.068008 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.068351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.068149 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.068351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.068322 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.070372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.070352 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.070544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.070524 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.124413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.124386 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.125010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.124957 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbft\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.253517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.253420 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:48.255122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.255086 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:48.255261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.255143 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:48.268231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.268210 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:48.717116 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:16:48.717049 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c271a6d_5a7a_4ca5_a273_845c84488ef5.slice/crio-c3c8c0b83e322b29f2fb09a3875af40a19667a9656a73220d55a68c99e768478 WatchSource:0}: Error finding container c3c8c0b83e322b29f2fb09a3875af40a19667a9656a73220d55a68c99e768478: Status 404 returned error can't find the container with id c3c8c0b83e322b29f2fb09a3875af40a19667a9656a73220d55a68c99e768478 Nov 04 01:16:48.718153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.718129 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:16:48.719423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.719391 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:48.719509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.719451 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:48.719509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.719481 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:16:48.961351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.961315 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" event={"ID":"1c271a6d-5a7a-4ca5-a273-845c84488ef5","Type":"ContainerStarted","Data":"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d"} Nov 04 01:16:48.961847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:48.961373 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" event={"ID":"1c271a6d-5a7a-4ca5-a273-845c84488ef5","Type":"ContainerStarted","Data":"c3c8c0b83e322b29f2fb09a3875af40a19667a9656a73220d55a68c99e768478"} Nov 04 01:16:49.071730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.071666 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podStartSLOduration=2.071649444 podStartE2EDuration="2.071649444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:49.071432976 +0000 UTC m=+1177.830095151" watchObservedRunningTime="2026-11-04 01:16:49.071649444 +0000 UTC m=+1177.830311606" Nov 04 01:16:49.254819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.254400 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:49.254819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.254459 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:49.268656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.268582 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:49.270747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.270722 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:49.270888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:49.270770 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:50.254006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:50.253393 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:50.254006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:50.253459 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:50.272652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:50.272618 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:50.272931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:50.272902 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:51.255367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:51.255329 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:51.255943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:51.255902 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:51.275367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:51.275336 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:51.275573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:51.275544 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:52.255389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:52.255348 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:52.255999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:52.255945 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:52.273324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:52.273297 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:52.273526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:52.273502 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:53.255436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:53.255392 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:53.256090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:53.256046 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:53.270437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:53.270395 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:53.270724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:53.270694 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:54.253178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:54.253137 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:16:54.253467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:54.253436 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 04 01:16:54.268910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:54.268866 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:54.269309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:54.268922 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:55.256563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:55.256519 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:55.256827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:55.256798 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:55.257559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:55.257539 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:16:55.268847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:55.268808 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 04 01:16:55.268994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:55.268857 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 04 01:16:56.713724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:56.713688 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:56.714881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:56.714864 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:56.728706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:56.728683 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:16:59.669573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.669533 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:16:59.674667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.674636 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.724552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.724515 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e993d7\"/\"distinct-w0-e993d7-kserve-self-signed-certs\"" Nov 04 01:16:59.726144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.726118 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:16:59.755235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755209 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.755353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755253 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.755353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755331 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.755483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755463 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xktj\" (UniqueName: \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.755565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755550 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.755658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.755643 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856328 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856361 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856386 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856572 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xktj\" (UniqueName: \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856643 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856672 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856757 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.856845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856817 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.857018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.856995 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.858783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.858760 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.858885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.858795 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.911057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.911028 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xktj\" (UniqueName: \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\") pod \"distinct-w0-e993d7-kserve-774f76c846-v2frl\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:16:59.987609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:16:59.987532 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:17:00.402218 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:17:00.402190 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a7f46ba_88fe_4622_b91a_07c42836b76a.slice/crio-74572a0d93ae5bbf704fff3c58436ea9f9d8de411d45b07b52731ebe67a82c9a WatchSource:0}: Error finding container 74572a0d93ae5bbf704fff3c58436ea9f9d8de411d45b07b52731ebe67a82c9a: Status 404 returned error can't find the container with id 74572a0d93ae5bbf704fff3c58436ea9f9d8de411d45b07b52731ebe67a82c9a Nov 04 01:17:00.403738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:00.403577 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:17:01.149797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:01.149757 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" event={"ID":"0a7f46ba-88fe-4622-b91a-07c42836b76a","Type":"ContainerStarted","Data":"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b"} Nov 04 01:17:01.150216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:01.149806 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" event={"ID":"0a7f46ba-88fe-4622-b91a-07c42836b76a","Type":"ContainerStarted","Data":"74572a0d93ae5bbf704fff3c58436ea9f9d8de411d45b07b52731ebe67a82c9a"} Nov 04 01:17:01.262465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:01.262402 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podStartSLOduration=2.2623887 podStartE2EDuration="2.2623887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:01.258601441 +0000 UTC m=+1190.017263601" watchObservedRunningTime="2026-11-04 01:17:01.2623887 +0000 UTC m=+1190.021050890" Nov 04 01:17:01.988578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:01.988542 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:17:09.398931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.398898 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:17:09.403552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.403533 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.456403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.456366 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:17:09.467618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.467582 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.467773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.467632 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xkk\" (UniqueName: \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.467845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.467771 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.467954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.467930 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.468028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.468003 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.468197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.468176 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.497924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.497898 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"kube-root-ca.crt\"" Nov 04 01:17:09.498126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.497911 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"openshift-service-ca.crt\"" Nov 04 01:17:09.500173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.500132 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"default-dockercfg-wmh77\"" Nov 04 01:17:09.500173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.500164 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-678aaf\"/\"distinct-w6-678aaf-kserve-self-signed-certs\"" Nov 04 01:17:09.569564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569527 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.569766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569716 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.569851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569751 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xkk\" (UniqueName: \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.569851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569827 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.569966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569882 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.569966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.569919 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.570105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.570029 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.570344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.570303 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.572798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.572743 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.575146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.575125 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.575696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.575624 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.628629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.628589 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xkk\" (UniqueName: \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\") pod \"distinct-w6-678aaf-kserve-859f56994d-rqxqr\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.717278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.717209 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:09.987920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:09.987833 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:17:10.001158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.001132 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:17:10.088138 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:17:10.088106 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod900a9742_6ff6_4144_8d96_16b72f1065f8.slice/crio-f38da8c5800bf85e08ee51e25ab9688e9a06c6d148a030a1b43cba80e3dd0dc0 WatchSource:0}: Error finding container f38da8c5800bf85e08ee51e25ab9688e9a06c6d148a030a1b43cba80e3dd0dc0: Status 404 returned error can't find the container with id f38da8c5800bf85e08ee51e25ab9688e9a06c6d148a030a1b43cba80e3dd0dc0 Nov 04 01:17:10.089479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.089458 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:17:10.225820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.225788 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" event={"ID":"900a9742-6ff6-4144-8d96-16b72f1065f8","Type":"ContainerStarted","Data":"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6"} Nov 04 01:17:10.226022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.225837 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" event={"ID":"900a9742-6ff6-4144-8d96-16b72f1065f8","Type":"ContainerStarted","Data":"f38da8c5800bf85e08ee51e25ab9688e9a06c6d148a030a1b43cba80e3dd0dc0"} Nov 04 01:17:10.241522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.241459 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:17:10.337252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.337205 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podStartSLOduration=1.337193126 podStartE2EDuration="1.337193126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:10.334036045 +0000 UTC m=+1199.092698207" watchObservedRunningTime="2026-11-04 01:17:10.337193126 +0000 UTC m=+1199.095855287" Nov 04 01:17:10.717488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:10.717459 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:11.487630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:11.487603 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:17:11.500206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:11.500179 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:17:11.502905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:11.502880 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:17:11.515764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:11.515742 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:17:13.834831 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:17:13.834787 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:17:14.894202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.894173 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:17:14.898327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.898311 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:14.960006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.959963 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:17:14.994781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.994754 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"kube-root-ca.crt\"" Nov 04 01:17:14.994947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.994881 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"openshift-service-ca.crt\"" Nov 04 01:17:14.995021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.994978 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"default-dockercfg-mblzv\"" Nov 04 01:17:14.995870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:14.995850 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"distinct-w5-e210ef-kserve-self-signed-certs\"" Nov 04 01:17:15.045947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.045887 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.046116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.045962 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.046217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.046200 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.046268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.046260 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.046363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.046339 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.046404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.046375 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889fr\" (UniqueName: \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147623 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147540 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147651 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147700 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147721 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147825 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889fr\" (UniqueName: \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.147964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.147952 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.148119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.148040 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.148119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.148094 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.148359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.148328 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.149975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.149955 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.150368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.150351 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.204318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.204298 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889fr\" (UniqueName: \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\") pod \"distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.212109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.212086 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:15.583490 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:17:15.583455 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1fb6f25_ef8f_4d1a_ba0b_5efedab64443.slice/crio-416857b17ae7bfd5737c5ccb150fccdd12ff21ee8c4fd9fa866550697a57a03c WatchSource:0}: Error finding container 416857b17ae7bfd5737c5ccb150fccdd12ff21ee8c4fd9fa866550697a57a03c: Status 404 returned error can't find the container with id 416857b17ae7bfd5737c5ccb150fccdd12ff21ee8c4fd9fa866550697a57a03c Nov 04 01:17:15.584703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:15.584678 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:17:16.279200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:16.279163 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" event={"ID":"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443","Type":"ContainerStarted","Data":"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c"} Nov 04 01:17:16.279582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:16.279216 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" event={"ID":"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443","Type":"ContainerStarted","Data":"416857b17ae7bfd5737c5ccb150fccdd12ff21ee8c4fd9fa866550697a57a03c"} Nov 04 01:17:16.391257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:16.391207 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podStartSLOduration=2.391193897 podStartE2EDuration="2.391193897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:16.388912129 +0000 UTC m=+1205.147574293" watchObservedRunningTime="2026-11-04 01:17:16.391193897 +0000 UTC m=+1205.149856057" Nov 04 01:17:17.212635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:17.212599 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:19.718292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:19.718262 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:19.731116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:19.731089 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:20.331199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:20.331163 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:17:25.212661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:25.212578 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:25.225136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:25.225112 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:25.372035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:25.372007 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:17:33.277854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.277814 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:17:33.283302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.283280 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.338935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.338903 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:17:33.377771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.377743 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"kube-root-ca.crt\"" Nov 04 01:17:33.377924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.377791 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"openshift-service-ca.crt\"" Nov 04 01:17:33.377924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.377814 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"default-dockercfg-f4xf6\"" Nov 04 01:17:33.385074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.385055 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"distinct-w1-c563ee-kserve-self-signed-certs\"" Nov 04 01:17:33.430846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.430804 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjtnh\" (UniqueName: \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.431059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.430925 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.431059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.430966 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.431212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.431093 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.431212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.431164 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.431212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.431187 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532319 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532386 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532498 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532570 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532592 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532685 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjtnh\" (UniqueName: \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.532940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.532905 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.533188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.533013 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.533369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.533332 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.535025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.535002 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.535136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.535086 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.589795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.589771 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjtnh\" (UniqueName: \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\") pod \"distinct-w1-c563ee-kserve-5d696b7c49-2jrgj\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.596609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.596584 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:33.971478 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:17:33.971447 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ec57e6_6f7b_4cb0_80a4_a3cb0263d61e.slice/crio-2bc2689eaf616242420300a0c2dda1c537ea056f448f7e9ec42af260678b7c3e WatchSource:0}: Error finding container 2bc2689eaf616242420300a0c2dda1c537ea056f448f7e9ec42af260678b7c3e: Status 404 returned error can't find the container with id 2bc2689eaf616242420300a0c2dda1c537ea056f448f7e9ec42af260678b7c3e Nov 04 01:17:33.972045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:33.972021 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:17:34.474922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:34.474879 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" event={"ID":"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e","Type":"ContainerStarted","Data":"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f"} Nov 04 01:17:34.475437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:34.474931 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" event={"ID":"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e","Type":"ContainerStarted","Data":"2bc2689eaf616242420300a0c2dda1c537ea056f448f7e9ec42af260678b7c3e"} Nov 04 01:17:34.586442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:34.586388 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podStartSLOduration=1.5863740769999999 podStartE2EDuration="1.586374077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:34.58565876 +0000 UTC m=+1223.344320922" watchObservedRunningTime="2026-11-04 01:17:34.586374077 +0000 UTC m=+1223.345036238" Nov 04 01:17:34.596713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:34.596686 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:43.597544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:43.597510 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:43.610627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:43.610601 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:17:43.831824 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:17:43.831785 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:17:44.569432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:17:44.569405 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:18:03.897083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:03.897049 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:18:03.902400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:03.902374 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:03.998891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:03.998854 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:18:04.013696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013662 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.013849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013706 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.013849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013832 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.013930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013889 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkwv\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.013994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013940 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.013994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.013962 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.014088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.014075 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.014123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.014100 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.014192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.014180 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115250 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115219 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115283 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115338 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115357 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115420 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115477 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkwv\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115515 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115546 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115601 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.115913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115882 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.116006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115929 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.116080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.115976 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.116080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.116070 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.116425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.116402 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.117754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.117734 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.117884 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.117867 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.169963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.169903 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.170215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.170194 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkwv\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.217366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.217340 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:04.647102 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:18:04.647072 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73edaad7_e2b6_4b23_89b6_c8d78566dbdf.slice/crio-7627d0b13ac1a5e77dffa7938510b455d8728e058f2325fe2a7bf6b1cd3a80d8 WatchSource:0}: Error finding container 7627d0b13ac1a5e77dffa7938510b455d8728e058f2325fe2a7bf6b1cd3a80d8: Status 404 returned error can't find the container with id 7627d0b13ac1a5e77dffa7938510b455d8728e058f2325fe2a7bf6b1cd3a80d8 Nov 04 01:18:04.648213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.648181 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:18:04.649788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.649756 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:18:04.649882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.649821 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:18:04.649882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.649850 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:18:04.751670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.751634 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" event={"ID":"73edaad7-e2b6-4b23-89b6-c8d78566dbdf","Type":"ContainerStarted","Data":"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11"} Nov 04 01:18:04.751871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.751696 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" event={"ID":"73edaad7-e2b6-4b23-89b6-c8d78566dbdf","Type":"ContainerStarted","Data":"7627d0b13ac1a5e77dffa7938510b455d8728e058f2325fe2a7bf6b1cd3a80d8"} Nov 04 01:18:04.864544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:04.864481 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podStartSLOduration=1.864463929 podStartE2EDuration="1.864463929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:04.863918863 +0000 UTC m=+1253.622581027" watchObservedRunningTime="2026-11-04 01:18:04.864463929 +0000 UTC m=+1253.623126091" Nov 04 01:18:05.218351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:05.218314 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:05.220264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:05.220239 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:05.220395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:05.220290 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:06.218061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:06.218029 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:06.218238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:06.218083 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:07.221359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:07.221320 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:07.221760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:07.221391 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:08.217808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:08.217772 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:08.218018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:08.217824 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:09.218221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:09.218185 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:09.218613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:09.218245 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:10.218360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:10.218325 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:10.218752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:10.218391 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:11.217729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:11.217690 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:18:11.217912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:11.217758 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 04 01:18:12.222010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:12.221967 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:12.222473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:12.222455 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:12.223218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:12.223198 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:18:13.833137 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:18:13.833099 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:18:43.834120 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:18:43.834069 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:18:51.213196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.213161 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h"] Nov 04 01:18:51.217733 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.217714 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.280637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.280607 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h"] Nov 04 01:18:51.311723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.311679 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:18:51.311872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.311739 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:18:51.333638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.333610 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.333778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.333748 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.333870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.333852 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntt9r\" (UniqueName: \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.435551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.435520 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.435730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.435579 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntt9r\" (UniqueName: \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.435730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.435724 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.436581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.436551 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.438025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.437978 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.490022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.489947 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntt9r\" (UniqueName: \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\") pod \"maas-api-revoke-sub-f4c50093-j2p5h\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.531573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.531546 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:51.918373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:51.918342 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h"] Nov 04 01:18:52.160648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:52.160615 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" event={"ID":"e4df72c4-7ee5-4b09-abcf-1ecd87359b06","Type":"ContainerStarted","Data":"ebd6dde6364caacf5b8f16f78ab8fe6ae501191584910e1b6339608e4e018fb2"} Nov 04 01:18:53.171222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:53.171180 4486 generic.go:356] "Generic (PLEG): container finished" podID="e4df72c4-7ee5-4b09-abcf-1ecd87359b06" containerID="37ff6433b1f99b1308ad478e5b077440220678d470d73d3ea5b0d58a997ce73c" exitCode=0 Nov 04 01:18:53.171683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:53.171270 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" event={"ID":"e4df72c4-7ee5-4b09-abcf-1ecd87359b06","Type":"ContainerDied","Data":"37ff6433b1f99b1308ad478e5b077440220678d470d73d3ea5b0d58a997ce73c"} Nov 04 01:18:54.341069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.341045 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:54.389891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.389853 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\") pod \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " Nov 04 01:18:54.390104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.390060 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\") pod \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " Nov 04 01:18:54.390205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.390190 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\" (UniqueName: \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\") pod \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\" (UID: \"e4df72c4-7ee5-4b09-abcf-1ecd87359b06\") " Nov 04 01:18:54.390289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.390259 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca" pod "e4df72c4-7ee5-4b09-abcf-1ecd87359b06" (UID: "e4df72c4-7ee5-4b09-abcf-1ecd87359b06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:18:54.390626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.390606 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:54.392721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.392693 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca" pod "e4df72c4-7ee5-4b09-abcf-1ecd87359b06" (UID: "e4df72c4-7ee5-4b09-abcf-1ecd87359b06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:18:54.392835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.392757 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r" pod "e4df72c4-7ee5-4b09-abcf-1ecd87359b06" (UID: "e4df72c4-7ee5-4b09-abcf-1ecd87359b06"). InnerVolumeSpecName "kube-api-access-ntt9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:18:54.491842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.491766 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:54.491842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:54.491796 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntt9r\" (UniqueName: \"kubernetes.io/projected/e4df72c4-7ee5-4b09-abcf-1ecd87359b06-kube-api-access-ntt9r\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:18:55.191303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:55.191272 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" Nov 04 01:18:55.191486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:55.191316 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h" event={"ID":"e4df72c4-7ee5-4b09-abcf-1ecd87359b06","Type":"ContainerDied","Data":"ebd6dde6364caacf5b8f16f78ab8fe6ae501191584910e1b6339608e4e018fb2"} Nov 04 01:18:55.191486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:18:55.191371 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd6dde6364caacf5b8f16f78ab8fe6ae501191584910e1b6339608e4e018fb2" Nov 04 01:19:05.455487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.455406 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz"] Nov 04 01:19:05.456027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.456011 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e4df72c4-7ee5-4b09-abcf-1ecd87359b06" containerName="revoke-keys" Nov 04 01:19:05.456093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.456027 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4df72c4-7ee5-4b09-abcf-1ecd87359b06" containerName="revoke-keys" Nov 04 01:19:05.456175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.456165 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4df72c4-7ee5-4b09-abcf-1ecd87359b06" containerName="revoke-keys" Nov 04 01:19:05.459663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.459640 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.513322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.513291 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:19:05.513600 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.513581 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:19:05.559169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.559129 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk5q\" (UniqueName: \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.559368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.559262 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.559460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.559434 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.569232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.569198 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz"] Nov 04 01:19:05.661223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.661189 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.661415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.661312 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk5q\" (UniqueName: \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.661415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.661368 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.662054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.662034 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.663639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.663610 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.718583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.718511 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk5q\" (UniqueName: \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\") pod \"maas-api-revoke-sub-7a9f57b4-v2bcz\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:05.774050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:05.773974 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:06.155976 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:19:06.155937 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7049a_bca5_4750_836d_48b5adb431c8.slice/crio-cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6 WatchSource:0}: Error finding container cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6: Status 404 returned error can't find the container with id cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6 Nov 04 01:19:06.158445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:06.158414 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz"] Nov 04 01:19:06.291269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:06.291235 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" event={"ID":"00f7049a-bca5-4750-836d-48b5adb431c8","Type":"ContainerStarted","Data":"cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6"} Nov 04 01:19:07.300504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:07.300419 4486 generic.go:356] "Generic (PLEG): container finished" podID="00f7049a-bca5-4750-836d-48b5adb431c8" containerID="d274a434420e3a124708a7b9b12cbdf6aa2f848e0a33956e0eb52aa05bc0ff5a" exitCode=0 Nov 04 01:19:07.300871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:07.300505 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" event={"ID":"00f7049a-bca5-4750-836d-48b5adb431c8","Type":"ContainerDied","Data":"d274a434420e3a124708a7b9b12cbdf6aa2f848e0a33956e0eb52aa05bc0ff5a"} Nov 04 01:19:08.459372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.459342 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:08.598948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.598911 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\" (UniqueName: \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\") pod \"00f7049a-bca5-4750-836d-48b5adb431c8\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " Nov 04 01:19:08.599153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.599021 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\") pod \"00f7049a-bca5-4750-836d-48b5adb431c8\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " Nov 04 01:19:08.599153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.599112 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\") pod \"00f7049a-bca5-4750-836d-48b5adb431c8\" (UID: \"00f7049a-bca5-4750-836d-48b5adb431c8\") " Nov 04 01:19:08.599760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.599724 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca" pod "00f7049a-bca5-4750-836d-48b5adb431c8" (UID: "00f7049a-bca5-4750-836d-48b5adb431c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:08.601552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.601505 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca" pod "00f7049a-bca5-4750-836d-48b5adb431c8" (UID: "00f7049a-bca5-4750-836d-48b5adb431c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:08.621168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.621137 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q" pod "00f7049a-bca5-4750-836d-48b5adb431c8" (UID: "00f7049a-bca5-4750-836d-48b5adb431c8"). InnerVolumeSpecName "kube-api-access-ljk5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:08.701467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.701437 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:08.701467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.701466 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk5q\" (UniqueName: \"kubernetes.io/projected/00f7049a-bca5-4750-836d-48b5adb431c8-kube-api-access-ljk5q\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:08.701646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:08.701477 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7049a-bca5-4750-836d-48b5adb431c8-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:09.318220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:09.318190 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" Nov 04 01:19:09.318220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:09.318212 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz" event={"ID":"00f7049a-bca5-4750-836d-48b5adb431c8","Type":"ContainerDied","Data":"cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6"} Nov 04 01:19:09.318430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:09.318259 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc8410a3406b3b8dfc53261399bee60b67d761553741b81fa623038abee25c6" Nov 04 01:19:13.831760 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:19:13.831722 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:19:16.067108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:16.067070 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm"] Nov 04 01:19:16.110765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:16.110732 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38779493-kdrjm"] Nov 04 01:19:17.350857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:17.350816 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbcf9e2-ec91-4b3b-9478-c411f217c56b" path="/var/lib/kubelet/pods/1dbcf9e2-ec91-4b3b-9478-c411f217c56b/volumes" Nov 04 01:19:27.055296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:27.055258 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst"] Nov 04 01:19:27.101687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:27.101658 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbd59ee-4wdst"] Nov 04 01:19:27.353068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:27.353029 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bacd9545-7571-487f-8c54-cbe0babd07ae" path="/var/lib/kubelet/pods/bacd9545-7571-487f-8c54-cbe0babd07ae/volumes" Nov 04 01:19:33.005644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.005609 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8"] Nov 04 01:19:33.006256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.006240 4486 cpu_manager.go:414] "containerMap: removing container" podUID="00f7049a-bca5-4750-836d-48b5adb431c8" containerName="revoke-keys" Nov 04 01:19:33.006306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.006256 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f7049a-bca5-4750-836d-48b5adb431c8" containerName="revoke-keys" Nov 04 01:19:33.006370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.006361 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f7049a-bca5-4750-836d-48b5adb431c8" containerName="revoke-keys" Nov 04 01:19:33.009970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.009945 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.079975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.079936 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8"] Nov 04 01:19:33.098382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.098355 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpz5\" (UniqueName: \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.098519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.098403 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.098519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.098419 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.107360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.107329 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:19:33.107360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.107349 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:19:33.200077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.200043 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.200202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.200084 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.200263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.200233 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpz5\" (UniqueName: \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.200711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.200688 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.202394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.202367 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.256143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.256089 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpz5\" (UniqueName: \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\") pod \"maas-api-revoke-sub-9a738a44-9trt8\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.324105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.324075 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:33.703007 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:19:33.702961 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd435d_8b8c_4513_a1bd_c6c96120e4d9.slice/crio-3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214 WatchSource:0}: Error finding container 3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214: Status 404 returned error can't find the container with id 3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214 Nov 04 01:19:33.704440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:33.704417 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8"] Nov 04 01:19:34.531572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:34.531539 4486 generic.go:356] "Generic (PLEG): container finished" podID="a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" containerID="7e1ac79498d1df70f9cb2d21f352237b0db73108e02720dae3ae27abd6b7609d" exitCode=0 Nov 04 01:19:34.531935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:34.531626 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" event={"ID":"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9","Type":"ContainerDied","Data":"7e1ac79498d1df70f9cb2d21f352237b0db73108e02720dae3ae27abd6b7609d"} Nov 04 01:19:34.531935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:34.531693 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" event={"ID":"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9","Type":"ContainerStarted","Data":"3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214"} Nov 04 01:19:35.693704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.693676 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:35.830951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.830924 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\") pod \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " Nov 04 01:19:35.831161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.831135 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\" (UniqueName: \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\") pod \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " Nov 04 01:19:35.831276 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.831184 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\") pod \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\" (UID: \"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9\") " Nov 04 01:19:35.832101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.832057 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca" pod "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" (UID: "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:35.834113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.834091 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca" pod "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" (UID: "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:35.834219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.834128 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5" pod "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" (UID: "a9cd435d-8b8c-4513-a1bd-c6c96120e4d9"). InnerVolumeSpecName "kube-api-access-5xpz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:35.932783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.932750 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpz5\" (UniqueName: \"kubernetes.io/projected/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-kube-api-access-5xpz5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:35.932783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.932779 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:35.932908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:35.932793 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:36.555613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:36.555582 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" Nov 04 01:19:36.555813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:36.555608 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8" event={"ID":"a9cd435d-8b8c-4513-a1bd-c6c96120e4d9","Type":"ContainerDied","Data":"3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214"} Nov 04 01:19:36.555813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:36.555670 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6beca6acf4f4b6ff9d18ae538a48b6b72f92d211a2d94574ebd50a90f17214" Nov 04 01:19:37.048056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:37.048017 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4"] Nov 04 01:19:37.052431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:37.052400 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a2927b-c9vc4"] Nov 04 01:19:37.350478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:37.350443 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ee568c-3982-4bea-9628-85b39af42195" path="/var/lib/kubelet/pods/73ee568c-3982-4bea-9628-85b39af42195/volumes" Nov 04 01:19:42.048591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:42.048555 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn"] Nov 04 01:19:42.095429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:42.095390 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6b4fa56-9mfhn"] Nov 04 01:19:43.350842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:43.350808 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b75b13-0217-45cf-97b0-32d0ccec919e" path="/var/lib/kubelet/pods/59b75b13-0217-45cf-97b0-32d0ccec919e/volumes" Nov 04 01:19:43.831502 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:19:43.831470 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:19:44.607182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.607147 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:19:44.608114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.608092 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" containerName="revoke-keys" Nov 04 01:19:44.608114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.608111 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" containerName="revoke-keys" Nov 04 01:19:44.608315 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.608262 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" containerName="revoke-keys" Nov 04 01:19:44.613556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.613535 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.666793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.666750 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.667129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.667098 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.667268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.667159 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.667268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.667231 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.667383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.667303 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.667439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.667385 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76h7\" (UniqueName: \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.674624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.674595 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:19:44.707245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.707211 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e210ef\"/\"distinct-2-w5-e210ef-kserve-self-signed-certs\"" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.768671 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.768790 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76h7\" (UniqueName: \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.768833 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.769014 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.769066 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.776004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.769243 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.782135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.781955 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.783713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.782851 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.793155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.788469 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.794713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.794681 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.794934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.794894 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.839400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.839374 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76h7\" (UniqueName: \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\") pod \"distinct-2-w5-e210ef-kserve-664795465d-5856z\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:44.926596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:44.926529 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:45.304053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:45.304026 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:19:45.304390 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:19:45.304369 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c96b0f2_11e4_444e_b064_9ef968de5399.slice/crio-fe2abcdae7c306401913654add87da2e6116fd8b5216d33dfd4f92eb55046ce9 WatchSource:0}: Error finding container fe2abcdae7c306401913654add87da2e6116fd8b5216d33dfd4f92eb55046ce9: Status 404 returned error can't find the container with id fe2abcdae7c306401913654add87da2e6116fd8b5216d33dfd4f92eb55046ce9 Nov 04 01:19:45.664256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:45.664213 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" event={"ID":"3c96b0f2-11e4-444e-b064-9ef968de5399","Type":"ContainerStarted","Data":"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71"} Nov 04 01:19:45.664256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:45.664265 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" event={"ID":"3c96b0f2-11e4-444e-b064-9ef968de5399","Type":"ContainerStarted","Data":"fe2abcdae7c306401913654add87da2e6116fd8b5216d33dfd4f92eb55046ce9"} Nov 04 01:19:45.780039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:45.779975 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podStartSLOduration=1.779962987 podStartE2EDuration="1.779962987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:45.774638484 +0000 UTC m=+1354.533300645" watchObservedRunningTime="2026-11-04 01:19:45.779962987 +0000 UTC m=+1354.538625208" Nov 04 01:19:45.928959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:45.927658 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:48.052041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.052007 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q"] Nov 04 01:19:48.096082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.096047 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21cf5835-p5m6q"] Nov 04 01:19:48.403757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.403725 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:19:48.408149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.408126 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.487693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.487659 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:19:48.516923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.516887 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.516939 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517066 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517145 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b9v\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517220 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517247 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517299 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517340 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.517520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.517466 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618610 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618640 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618818 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618870 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618915 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.618965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.618945 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619000 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b9v\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619082 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619086 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619114 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619153 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619225 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619442 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.619510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.619489 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.620947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.620924 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.621533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.621516 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.674613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.674556 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b9v\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.674613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.674584 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.722545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.722521 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:48.805976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.805946 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd"] Nov 04 01:19:48.811669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.811648 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:48.869923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.862235 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:19:48.869923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.862716 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:19:48.869923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.866548 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd"] Nov 04 01:19:48.925941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.925864 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:48.925941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.925911 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:48.926210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:48.926184 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7nqw\" (UniqueName: \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.027573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.027546 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7nqw\" (UniqueName: \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.027699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.027652 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.027699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.027680 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.028374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.028352 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.029876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.029856 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.083149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.083120 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7nqw\" (UniqueName: \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\") pod \"maas-api-revoke-sub-f0fd8de4-mccwd\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.130340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.130312 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:49.147629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.147601 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:19:49.150940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.150910 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:19:49.151061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.150972 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:19:49.151116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.151056 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:19:49.350022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.349973 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97644f5-d5d5-435b-a2ed-e493e82e99c7" path="/var/lib/kubelet/pods/f97644f5-d5d5-435b-a2ed-e493e82e99c7/volumes" Nov 04 01:19:49.510337 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:19:49.510305 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4acfb06_4ed1_4b6b_b8bc_cf9b71cdbebe.slice/crio-e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8 WatchSource:0}: Error finding container e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8: Status 404 returned error can't find the container with id e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8 Nov 04 01:19:49.511145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.511124 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd"] Nov 04 01:19:49.698466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.698424 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" event={"ID":"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe","Type":"ContainerStarted","Data":"e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8"} Nov 04 01:19:49.700107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.700080 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" event={"ID":"0c58f026-1d30-4188-9bb0-251c1a968b20","Type":"ContainerStarted","Data":"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406"} Nov 04 01:19:49.700274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.700129 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" event={"ID":"0c58f026-1d30-4188-9bb0-251c1a968b20","Type":"ContainerStarted","Data":"85e773ddbfae0d78df712d2477d81d2d887741a5d762eae52b8cee65d068fa03"} Nov 04 01:19:49.723126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.723105 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:49.822856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:49.822800 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podStartSLOduration=1.8227814919999998 podStartE2EDuration="1.822781492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:49.817725674 +0000 UTC m=+1358.576387834" watchObservedRunningTime="2026-11-04 01:19:49.822781492 +0000 UTC m=+1358.581443653" Nov 04 01:19:50.709372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:50.709341 4486 generic.go:356] "Generic (PLEG): container finished" podID="c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" containerID="17d2b3d5f6545aebb47be95848802390a0ea4f94c0b5cefbe2da6c3f1a55c353" exitCode=0 Nov 04 01:19:50.709760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:50.709411 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" event={"ID":"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe","Type":"ContainerDied","Data":"17d2b3d5f6545aebb47be95848802390a0ea4f94c0b5cefbe2da6c3f1a55c353"} Nov 04 01:19:50.724907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:50.724880 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:50.725024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:50.724920 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:51.723518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:51.723487 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:51.723920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:51.723546 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:52.001899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.001503 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:52.092393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.092361 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\") pod \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " Nov 04 01:19:52.092573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.092462 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\") pod \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " Nov 04 01:19:52.092652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.092583 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\" (UniqueName: \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\") pod \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\" (UID: \"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe\") " Nov 04 01:19:52.092828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.092804 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca" pod "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" (UID: "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:19:52.093015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.092968 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.094418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.094392 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca" pod "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" (UID: "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:19:52.094553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.094533 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw" pod "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" (UID: "c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe"). InnerVolumeSpecName "kube-api-access-x7nqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:19:52.209005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.203019 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7nqw\" (UniqueName: \"kubernetes.io/projected/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-kube-api-access-x7nqw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.209005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.203061 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:19:52.723614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.723583 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:52.724020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.723630 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:52.727456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.727428 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" event={"ID":"c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe","Type":"ContainerDied","Data":"e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8"} Nov 04 01:19:52.727575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.727505 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd" Nov 04 01:19:52.727575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:52.727510 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25af89ddbe63f2e127456c861152251df4f94f17aec3b856cc65b14dd7647f8" Nov 04 01:19:53.723709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:53.723672 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:53.724113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:53.723736 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:54.722824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:54.722791 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:54.723009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:54.722855 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:54.927291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:54.927264 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:54.942619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:54.942592 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:55.723666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:55.723631 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:19:55.723833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:55.723702 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 04 01:19:55.796803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:55.796770 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:19:56.727387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:56.727356 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:56.727879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:56.727860 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:56.728667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:56.728646 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:19:57.142189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.142152 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:19:57.142878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.142857 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" containerName="revoke-keys" Nov 04 01:19:57.142977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.142879 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" containerName="revoke-keys" Nov 04 01:19:57.143077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.143051 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" containerName="revoke-keys" Nov 04 01:19:57.146967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.146946 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.197996 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.197946 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c563ee\"/\"distinct-2-w1-c563ee-kserve-self-signed-certs\"" Nov 04 01:19:57.201425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201392 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.201547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201445 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.201594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201544 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.201594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201575 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.201699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201619 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.201753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.201725 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bj5j\" (UniqueName: \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.203887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.203863 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:19:57.303360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.303319 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.304161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.304126 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.304293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.304198 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.304361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.304341 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bj5j\" (UniqueName: \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.305701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.305679 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.305883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.305858 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.306127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.305373 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.306127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.304012 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.306324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.306291 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.311598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.311572 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.313589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.313567 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.366970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.366936 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bj5j\" (UniqueName: \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\") pod \"distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.460142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.460081 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:19:57.830583 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:19:57.830548 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83c095d_a66c_4f10_9f6c_cb3d6c2ff098.slice/crio-e944ab9565b1ac00547bc990dc083da5f726e4f59436fcd48cdf7b0cab75de5c WatchSource:0}: Error finding container e944ab9565b1ac00547bc990dc083da5f726e4f59436fcd48cdf7b0cab75de5c: Status 404 returned error can't find the container with id e944ab9565b1ac00547bc990dc083da5f726e4f59436fcd48cdf7b0cab75de5c Nov 04 01:19:57.831285 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:57.831176 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:19:58.807357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:58.807326 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" event={"ID":"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098","Type":"ContainerStarted","Data":"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61"} Nov 04 01:19:58.807533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:58.807371 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" event={"ID":"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098","Type":"ContainerStarted","Data":"e944ab9565b1ac00547bc990dc083da5f726e4f59436fcd48cdf7b0cab75de5c"} Nov 04 01:19:58.920875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:58.920816 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podStartSLOduration=1.9208000699999999 podStartE2EDuration="1.92080007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:58.916268892 +0000 UTC m=+1367.674931053" watchObservedRunningTime="2026-11-04 01:19:58.92080007 +0000 UTC m=+1367.679462230" Nov 04 01:19:59.460386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:19:59.460348 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:03.099377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.099342 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:20:03.103354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.103335 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.178098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.178060 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:20:03.180472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180434 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.180607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180487 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.180685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180606 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.180815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180795 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.180882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180856 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.180945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180884 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.181084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180946 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.181084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.180978 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.181084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.181039 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbw8\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.281939 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282012 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282061 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282093 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282142 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbw8\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282197 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282225 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282268 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282453 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282472 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282623 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282526 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282739 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282750 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.282912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.282891 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.284388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.284366 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.284611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.284592 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.339382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.339355 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.339560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.339538 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbw8\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.417655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.417574 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:03.522802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.522769 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:20:03.546164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.543553 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.587020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.586934 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:20:03.589226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589196 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp44f\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589286 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589321 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589393 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589450 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589701 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589761 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589852 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.589919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.589878 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691053 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691097 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691157 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691176 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691214 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp44f\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691266 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691296 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691363 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691421 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691743 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691765 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691878 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.691924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.691912 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.692279 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.692258 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.693463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.693440 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.693749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.693731 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.745813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.745786 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.745944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.745895 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp44f\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\") pod \"e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:03.849012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.848603 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:20:03.850268 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:20:03.850219 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3891318_4bb2_4c56_bea8_f6a314fe7fcb.slice/crio-f03a157fa06b8f938ec771188872d730cd2a979758d199774dbbc78f620e0201 WatchSource:0}: Error finding container f03a157fa06b8f938ec771188872d730cd2a979758d199774dbbc78f620e0201: Status 404 returned error can't find the container with id f03a157fa06b8f938ec771188872d730cd2a979758d199774dbbc78f620e0201 Nov 04 01:20:03.853330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.853290 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:03.853414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.853375 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:03.853453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.853417 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:03.863898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:03.863874 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:04.296268 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:20:04.296232 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93c59d3_186c_4e77_a8ae_40c1e4102cc4.slice/crio-3ac1fa4065552f5a1075cd293ee628017a62fdb7faaaac2dd4737749a1d447e0 WatchSource:0}: Error finding container 3ac1fa4065552f5a1075cd293ee628017a62fdb7faaaac2dd4737749a1d447e0: Status 404 returned error can't find the container with id 3ac1fa4065552f5a1075cd293ee628017a62fdb7faaaac2dd4737749a1d447e0 Nov 04 01:20:04.296690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.296499 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:20:04.299243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.299213 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:04.299339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.299272 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:04.299339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.299303 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:20:04.862607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.862567 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" event={"ID":"f3891318-4bb2-4c56-bea8-f6a314fe7fcb","Type":"ContainerStarted","Data":"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded"} Nov 04 01:20:04.862783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.862636 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" event={"ID":"f3891318-4bb2-4c56-bea8-f6a314fe7fcb","Type":"ContainerStarted","Data":"f03a157fa06b8f938ec771188872d730cd2a979758d199774dbbc78f620e0201"} Nov 04 01:20:04.864501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.864475 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" event={"ID":"d93c59d3-186c-4e77-a8ae-40c1e4102cc4","Type":"ContainerStarted","Data":"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb"} Nov 04 01:20:04.864632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:04.864515 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" event={"ID":"d93c59d3-186c-4e77-a8ae-40c1e4102cc4","Type":"ContainerStarted","Data":"3ac1fa4065552f5a1075cd293ee628017a62fdb7faaaac2dd4737749a1d447e0"} Nov 04 01:20:05.005701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.005634 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podStartSLOduration=2.005619084 podStartE2EDuration="2.005619084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:04.988348953 +0000 UTC m=+1373.747011115" watchObservedRunningTime="2026-11-04 01:20:05.005619084 +0000 UTC m=+1373.764281243" Nov 04 01:20:05.109542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.109480 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podStartSLOduration=2.109464871 podStartE2EDuration="2.109464871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:05.105121593 +0000 UTC m=+1373.863783753" watchObservedRunningTime="2026-11-04 01:20:05.109464871 +0000 UTC m=+1373.868127031" Nov 04 01:20:05.418707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.418631 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:05.420367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.420343 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:05.420564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.420397 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:05.888007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.887226 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:05.896005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.895267 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:05.896005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:05.895325 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:06.420012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:06.417822 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:06.420012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:06.417868 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:06.864515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:06.864481 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:06.864756 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:06.864729 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:07.418443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.418413 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:07.418669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.418650 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:07.465015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.464953 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:07.505157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.505128 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:07.871822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.871789 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:07.872122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.872090 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:07.985015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:07.983342 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:08.418452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:08.418413 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:08.418634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:08.418479 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:08.870639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:08.870598 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:08.871169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:08.870670 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:09.418266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:09.418234 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:09.418477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:09.418282 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:09.871004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:09.870293 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:09.871004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:09.870351 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:10.422025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:10.418308 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:20:10.422025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:10.418366 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 04 01:20:10.866590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:10.864647 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:20:10.866590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:10.864707 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 04 01:20:12.562006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.561516 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:20:12.574005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.571074 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:12.574005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.571148 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.585002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.580054 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.639006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.635088 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.794937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.794155 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:20:12.854614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.854573 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.885318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.885286 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:12.886490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.886154 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:12.886490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.886205 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:12.887995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.887960 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:12.890536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.890518 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:20:12.945901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:12.944912 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.229230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.229118 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.540833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.540733 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:13.542417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.542395 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:13.543664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.543647 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:20:13.577012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.574085 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.577012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.574653 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:20:13.584118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.584076 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.585102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.585076 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:20:13.661018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.650079 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.663490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.663464 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:20:13.848007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.847115 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:13.865057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.859949 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:20:13.985777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.985736 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:20:13.991858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.991808 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:20:13.992184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.992159 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:20:13.994500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.994468 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:20:13.995893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:13.995875 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:14.072029 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:20:14.067736 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:20:14.123180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:14.123101 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:20:14.133699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:14.133673 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:20:14.300013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:14.288576 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:20:14.318725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:14.318692 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:20:14.374077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:14.365947 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:20:15.198286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.198249 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77"] Nov 04 01:20:15.205409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.205383 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.228025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.227977 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw56\" (UniqueName: \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.228310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.228290 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.228472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.228456 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.261462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.255747 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:15.261462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.255910 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:20:15.261462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.257485 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77"] Nov 04 01:20:15.329718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.329687 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw56\" (UniqueName: \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.329961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.329937 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.330186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.330165 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.330696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.330671 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.333188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.333162 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.386072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.386037 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw56\" (UniqueName: \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\") pod \"maas-api-revoke-sub-4cb0c968-zxh77\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.523526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.523448 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:15.909133 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:20:15.909098 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf9b378_6f7e_4728_9c7b_8510d706898d.slice/crio-51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795 WatchSource:0}: Error finding container 51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795: Status 404 returned error can't find the container with id 51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795 Nov 04 01:20:15.910203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:15.910175 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77"] Nov 04 01:20:16.486349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:16.486307 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" event={"ID":"3bf9b378-6f7e-4728-9c7b-8510d706898d","Type":"ContainerStarted","Data":"8506bd5427be5e7c9b839c6baf295c0ec9bc5ca61f3cc7114550d62904fddaee"} Nov 04 01:20:16.486704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:16.486375 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" event={"ID":"3bf9b378-6f7e-4728-9c7b-8510d706898d","Type":"ContainerStarted","Data":"51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795"} Nov 04 01:20:16.603853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:16.603803 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" podStartSLOduration=1.603789977 podStartE2EDuration="1.603789977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:16.601685585 +0000 UTC m=+1385.360347750" watchObservedRunningTime="2026-11-04 01:20:16.603789977 +0000 UTC m=+1385.362452136" Nov 04 01:20:17.496716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:17.496677 4486 generic.go:356] "Generic (PLEG): container finished" podID="3bf9b378-6f7e-4728-9c7b-8510d706898d" containerID="8506bd5427be5e7c9b839c6baf295c0ec9bc5ca61f3cc7114550d62904fddaee" exitCode=0 Nov 04 01:20:17.497157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:17.496753 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" event={"ID":"3bf9b378-6f7e-4728-9c7b-8510d706898d","Type":"ContainerDied","Data":"8506bd5427be5e7c9b839c6baf295c0ec9bc5ca61f3cc7114550d62904fddaee"} Nov 04 01:20:18.678812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.678787 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:18.782396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.782318 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\") pod \"3bf9b378-6f7e-4728-9c7b-8510d706898d\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " Nov 04 01:20:18.782396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.782371 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\" (UniqueName: \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\") pod \"3bf9b378-6f7e-4728-9c7b-8510d706898d\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " Nov 04 01:20:18.782573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.782451 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\") pod \"3bf9b378-6f7e-4728-9c7b-8510d706898d\" (UID: \"3bf9b378-6f7e-4728-9c7b-8510d706898d\") " Nov 04 01:20:18.782822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.782782 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca" pod "3bf9b378-6f7e-4728-9c7b-8510d706898d" (UID: "3bf9b378-6f7e-4728-9c7b-8510d706898d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:18.783043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.783022 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:18.784310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.784290 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca" pod "3bf9b378-6f7e-4728-9c7b-8510d706898d" (UID: "3bf9b378-6f7e-4728-9c7b-8510d706898d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:18.784532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.784513 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56" pod "3bf9b378-6f7e-4728-9c7b-8510d706898d" (UID: "3bf9b378-6f7e-4728-9c7b-8510d706898d"). InnerVolumeSpecName "kube-api-access-fnw56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:18.885162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.885129 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf9b378-6f7e-4728-9c7b-8510d706898d-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:18.885162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:18.885165 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnw56\" (UniqueName: \"kubernetes.io/projected/3bf9b378-6f7e-4728-9c7b-8510d706898d-kube-api-access-fnw56\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:19.514620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:19.514590 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" Nov 04 01:20:19.514835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:19.514625 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77" event={"ID":"3bf9b378-6f7e-4728-9c7b-8510d706898d","Type":"ContainerDied","Data":"51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795"} Nov 04 01:20:19.514835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:19.514667 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51c466d75f1b9ab2f797138fe2965170a1ffc42c36279176ca54ea23eef82795" Nov 04 01:20:20.293612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.293577 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq"] Nov 04 01:20:20.294235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.294213 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3bf9b378-6f7e-4728-9c7b-8510d706898d" containerName="revoke-keys" Nov 04 01:20:20.294235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.294230 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf9b378-6f7e-4728-9c7b-8510d706898d" containerName="revoke-keys" Nov 04 01:20:20.294375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.294366 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf9b378-6f7e-4728-9c7b-8510d706898d" containerName="revoke-keys" Nov 04 01:20:20.298443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.298423 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.351879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.351854 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq"] Nov 04 01:20:20.393379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.393348 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:20.393521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.393380 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:20:20.402626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.402601 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.402626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.402629 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.403059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.402962 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jglwm\" (UniqueName: \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.504442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.504408 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.504442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.504444 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.504626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.504612 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jglwm\" (UniqueName: \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.505158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.505138 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.506661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.506644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.560832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.560770 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jglwm\" (UniqueName: \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\") pod \"maas-api-revoke-sub-88c615a2-9vwjq\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.612081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.612045 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:20.995397 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:20:20.995366 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71ddd67_1b6b_440c_bdeb_e2cc9f088459.slice/crio-52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a WatchSource:0}: Error finding container 52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a: Status 404 returned error can't find the container with id 52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a Nov 04 01:20:20.996441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:20.996410 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq"] Nov 04 01:20:21.051089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.051050 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m"] Nov 04 01:20:21.065422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.059624 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee9685b6-f557m"] Nov 04 01:20:21.352349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.352321 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91083279-ddc7-4738-bf50-088f75a01ab9" path="/var/lib/kubelet/pods/91083279-ddc7-4738-bf50-088f75a01ab9/volumes" Nov 04 01:20:21.535169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.535130 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" event={"ID":"c71ddd67-1b6b-440c-bdeb-e2cc9f088459","Type":"ContainerStarted","Data":"3532843b3f19a58c05a381fd7e327cf619de944c850ff63d084e96e4151b42ab"} Nov 04 01:20:21.535329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.535197 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" event={"ID":"c71ddd67-1b6b-440c-bdeb-e2cc9f088459","Type":"ContainerStarted","Data":"52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a"} Nov 04 01:20:21.645625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:21.645522 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" podStartSLOduration=1.6455074280000002 podStartE2EDuration="1.645507428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:21.644059652 +0000 UTC m=+1390.402721811" watchObservedRunningTime="2026-11-04 01:20:21.645507428 +0000 UTC m=+1390.404169635" Nov 04 01:20:22.547561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:22.547524 4486 generic.go:356] "Generic (PLEG): container finished" podID="c71ddd67-1b6b-440c-bdeb-e2cc9f088459" containerID="3532843b3f19a58c05a381fd7e327cf619de944c850ff63d084e96e4151b42ab" exitCode=0 Nov 04 01:20:22.548053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:22.547607 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" event={"ID":"c71ddd67-1b6b-440c-bdeb-e2cc9f088459","Type":"ContainerDied","Data":"3532843b3f19a58c05a381fd7e327cf619de944c850ff63d084e96e4151b42ab"} Nov 04 01:20:23.708151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.708121 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:23.849849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.849808 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\") pod \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " Nov 04 01:20:23.850125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.850104 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\" (UniqueName: \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\") pod \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " Nov 04 01:20:23.850247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.850165 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\") pod \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\" (UID: \"c71ddd67-1b6b-440c-bdeb-e2cc9f088459\") " Nov 04 01:20:23.850247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.850189 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca" pod "c71ddd67-1b6b-440c-bdeb-e2cc9f088459" (UID: "c71ddd67-1b6b-440c-bdeb-e2cc9f088459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:23.850878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.850852 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:23.852514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.852489 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca" pod "c71ddd67-1b6b-440c-bdeb-e2cc9f088459" (UID: "c71ddd67-1b6b-440c-bdeb-e2cc9f088459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:23.852769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.852739 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm" pod "c71ddd67-1b6b-440c-bdeb-e2cc9f088459" (UID: "c71ddd67-1b6b-440c-bdeb-e2cc9f088459"). InnerVolumeSpecName "kube-api-access-jglwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:23.952480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.952444 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jglwm\" (UniqueName: \"kubernetes.io/projected/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-kube-api-access-jglwm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:23.952480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:23.952476 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71ddd67-1b6b-440c-bdeb-e2cc9f088459-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:24.565886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:24.565858 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" Nov 04 01:20:24.566095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:24.565896 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq" event={"ID":"c71ddd67-1b6b-440c-bdeb-e2cc9f088459","Type":"ContainerDied","Data":"52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a"} Nov 04 01:20:24.566095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:24.565940 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ac192d7137d7862e1c055fe48fa7b3a17843e6679081d7de514cc69fb6970a" Nov 04 01:20:35.391913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.389280 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz"] Nov 04 01:20:35.391913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.390279 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c71ddd67-1b6b-440c-bdeb-e2cc9f088459" containerName="revoke-keys" Nov 04 01:20:35.391913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.390302 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71ddd67-1b6b-440c-bdeb-e2cc9f088459" containerName="revoke-keys" Nov 04 01:20:35.391913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.390501 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71ddd67-1b6b-440c-bdeb-e2cc9f088459" containerName="revoke-keys" Nov 04 01:20:35.396371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.395180 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.483641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.483615 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:35.492486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.492457 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:20:35.497728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.497702 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz"] Nov 04 01:20:35.498197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.498177 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thxnk\" (UniqueName: \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.498331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.498311 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.498479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.498464 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.599519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.599488 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.599669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.599586 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.599669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.599664 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thxnk\" (UniqueName: \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.600252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.600228 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.601744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.601725 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.654579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.654521 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thxnk\" (UniqueName: \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\") pod \"maas-api-revoke-sub-e17c15cb-5rnxz\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:35.712139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:35.712113 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:36.048754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.048679 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29"] Nov 04 01:20:36.053190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.053169 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf40af40-k2z29"] Nov 04 01:20:36.092265 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:20:36.092227 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea1ab2_3743_406b_8c44_aa929f147f1e.slice/crio-5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7 WatchSource:0}: Error finding container 5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7: Status 404 returned error can't find the container with id 5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7 Nov 04 01:20:36.093554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.093534 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz"] Nov 04 01:20:36.667450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.667355 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" event={"ID":"41ea1ab2-3743-406b-8c44-aa929f147f1e","Type":"ContainerStarted","Data":"bfa8ba4a6f52d695271a4df1803c6dd493f93b396d5a64850be817a76df2b8a5"} Nov 04 01:20:36.667450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.667422 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" event={"ID":"41ea1ab2-3743-406b-8c44-aa929f147f1e","Type":"ContainerStarted","Data":"5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7"} Nov 04 01:20:36.794956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:36.794904 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" podStartSLOduration=1.794890179 podStartE2EDuration="1.794890179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:36.790122132 +0000 UTC m=+1405.548784291" watchObservedRunningTime="2026-11-04 01:20:36.794890179 +0000 UTC m=+1405.553552338" Nov 04 01:20:37.048779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.048750 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz"] Nov 04 01:20:37.053592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.053564 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d624158f-4frkz"] Nov 04 01:20:37.354891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.354855 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884" path="/var/lib/kubelet/pods/60f2e88d-a3a0-45e7-a3c1-aba4eb6ff884/volumes" Nov 04 01:20:37.355342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.355327 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9641c0ff-fce2-40a1-9c76-efebfb61319a" path="/var/lib/kubelet/pods/9641c0ff-fce2-40a1-9c76-efebfb61319a/volumes" Nov 04 01:20:37.677200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.677111 4486 generic.go:356] "Generic (PLEG): container finished" podID="41ea1ab2-3743-406b-8c44-aa929f147f1e" containerID="bfa8ba4a6f52d695271a4df1803c6dd493f93b396d5a64850be817a76df2b8a5" exitCode=0 Nov 04 01:20:37.677612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:37.677197 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" event={"ID":"41ea1ab2-3743-406b-8c44-aa929f147f1e","Type":"ContainerDied","Data":"bfa8ba4a6f52d695271a4df1803c6dd493f93b396d5a64850be817a76df2b8a5"} Nov 04 01:20:38.059805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.059716 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j"] Nov 04 01:20:38.066886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.066859 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cca7975-9tw8j"] Nov 04 01:20:38.857082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.857053 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:38.941904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.941831 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\") pod \"41ea1ab2-3743-406b-8c44-aa929f147f1e\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " Nov 04 01:20:38.942081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.941928 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\") pod \"41ea1ab2-3743-406b-8c44-aa929f147f1e\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " Nov 04 01:20:38.942081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.941979 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\" (UniqueName: \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\") pod \"41ea1ab2-3743-406b-8c44-aa929f147f1e\" (UID: \"41ea1ab2-3743-406b-8c44-aa929f147f1e\") " Nov 04 01:20:38.942239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.942212 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca" pod "41ea1ab2-3743-406b-8c44-aa929f147f1e" (UID: "41ea1ab2-3743-406b-8c44-aa929f147f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:38.942634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.942596 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:38.943894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.943868 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca" pod "41ea1ab2-3743-406b-8c44-aa929f147f1e" (UID: "41ea1ab2-3743-406b-8c44-aa929f147f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:38.944023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:38.943953 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk" pod "41ea1ab2-3743-406b-8c44-aa929f147f1e" (UID: "41ea1ab2-3743-406b-8c44-aa929f147f1e"). InnerVolumeSpecName "kube-api-access-thxnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:39.044273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.044234 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41ea1ab2-3743-406b-8c44-aa929f147f1e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:39.044273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.044269 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thxnk\" (UniqueName: \"kubernetes.io/projected/41ea1ab2-3743-406b-8c44-aa929f147f1e-kube-api-access-thxnk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:39.350895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.350862 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119fa33a-8d9f-4d4d-a325-2ae21c5c5607" path="/var/lib/kubelet/pods/119fa33a-8d9f-4d4d-a325-2ae21c5c5607/volumes" Nov 04 01:20:39.695751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.695677 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" Nov 04 01:20:39.695751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.695709 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz" event={"ID":"41ea1ab2-3743-406b-8c44-aa929f147f1e","Type":"ContainerDied","Data":"5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7"} Nov 04 01:20:39.695919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:39.695758 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f644659f95be0961f611f37258fa3c24230a8c2e0b002f2d1bb4d6f9ce202e7" Nov 04 01:20:43.830242 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:20:43.830202 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:20:45.633591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.633552 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb"] Nov 04 01:20:45.634243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.634225 4486 cpu_manager.go:414] "containerMap: removing container" podUID="41ea1ab2-3743-406b-8c44-aa929f147f1e" containerName="revoke-keys" Nov 04 01:20:45.634243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.634242 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ea1ab2-3743-406b-8c44-aa929f147f1e" containerName="revoke-keys" Nov 04 01:20:45.634380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.634371 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ea1ab2-3743-406b-8c44-aa929f147f1e" containerName="revoke-keys" Nov 04 01:20:45.639323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.639301 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.695079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.695048 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb"] Nov 04 01:20:45.763146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.745224 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:45.786003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.771251 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:20:45.858515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.850201 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.858515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.850274 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.858515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.850316 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tlh\" (UniqueName: \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.960076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.955198 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.960905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.960876 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.966631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.961874 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tlh\" (UniqueName: \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.968479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.961793 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:45.994593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:45.994527 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:46.036660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:46.036564 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tlh\" (UniqueName: \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\") pod \"maas-api-revoke-sub-d83b6d4e-7cshb\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:46.253648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:46.253543 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:47.072488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:47.072456 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" event={"ID":"37aaabab-a61e-43a2-a92f-6b5a2517b1ab","Type":"ContainerStarted","Data":"dde8060d54d6c9fa5e933ae19b52dfd0df5a50d61fe76d8dae88d402024daada"} Nov 04 01:20:47.082254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:47.082185 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb"] Nov 04 01:20:47.384066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:47.368135 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:48.013102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:48.013058 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:48.137133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:48.136048 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:20:48.139341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:48.139314 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" event={"ID":"37aaabab-a61e-43a2-a92f-6b5a2517b1ab","Type":"ContainerStarted","Data":"3dba2746186e0892a78f33f7c7f0c488938ef233925071c743919a79ca1d2932"} Nov 04 01:20:48.178380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:48.178069 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:48.257121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:48.257070 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" podStartSLOduration=3.2570558800000002 podStartE2EDuration="3.25705588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:48.252582714 +0000 UTC m=+1417.011244873" watchObservedRunningTime="2026-11-04 01:20:48.25705588 +0000 UTC m=+1417.015718039" Nov 04 01:20:49.048403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:49.048366 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz"] Nov 04 01:20:49.096202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:49.096156 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e6de66-fblnz"] Nov 04 01:20:49.152261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:49.152225 4486 generic.go:356] "Generic (PLEG): container finished" podID="37aaabab-a61e-43a2-a92f-6b5a2517b1ab" containerID="3dba2746186e0892a78f33f7c7f0c488938ef233925071c743919a79ca1d2932" exitCode=0 Nov 04 01:20:49.152618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:49.152308 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" event={"ID":"37aaabab-a61e-43a2-a92f-6b5a2517b1ab","Type":"ContainerDied","Data":"3dba2746186e0892a78f33f7c7f0c488938ef233925071c743919a79ca1d2932"} Nov 04 01:20:49.350357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:49.350328 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7147e754-f7ab-4b97-91da-6a9df9b6cb06" path="/var/lib/kubelet/pods/7147e754-f7ab-4b97-91da-6a9df9b6cb06/volumes" Nov 04 01:20:50.344541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.344505 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:50.421715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.421641 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\") pod \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " Nov 04 01:20:50.421852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.421752 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\") pod \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " Nov 04 01:20:50.421931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.421914 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\" (UniqueName: \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\") pod \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\" (UID: \"37aaabab-a61e-43a2-a92f-6b5a2517b1ab\") " Nov 04 01:20:50.422080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.422059 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca" pod "37aaabab-a61e-43a2-a92f-6b5a2517b1ab" (UID: "37aaabab-a61e-43a2-a92f-6b5a2517b1ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:50.422584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.422565 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:50.424220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.424191 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca" pod "37aaabab-a61e-43a2-a92f-6b5a2517b1ab" (UID: "37aaabab-a61e-43a2-a92f-6b5a2517b1ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:50.424312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.424283 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh" pod "37aaabab-a61e-43a2-a92f-6b5a2517b1ab" (UID: "37aaabab-a61e-43a2-a92f-6b5a2517b1ab"). InnerVolumeSpecName "kube-api-access-b6tlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:50.524260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.524225 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tlh\" (UniqueName: \"kubernetes.io/projected/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-kube-api-access-b6tlh\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:50.524260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:50.524256 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aaabab-a61e-43a2-a92f-6b5a2517b1ab-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:51.175034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:51.174957 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" event={"ID":"37aaabab-a61e-43a2-a92f-6b5a2517b1ab","Type":"ContainerDied","Data":"dde8060d54d6c9fa5e933ae19b52dfd0df5a50d61fe76d8dae88d402024daada"} Nov 04 01:20:51.175201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:51.175043 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde8060d54d6c9fa5e933ae19b52dfd0df5a50d61fe76d8dae88d402024daada" Nov 04 01:20:51.175201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:51.174967 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb" Nov 04 01:20:53.073530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.073493 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs"] Nov 04 01:20:53.104712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.104673 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97"] Nov 04 01:20:53.111915 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.111891 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30128f0-4n4hs"] Nov 04 01:20:53.116726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.116702 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f832e-rwf97"] Nov 04 01:20:53.350380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.350307 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf70af16-418f-4fa5-a904-75988861ce47" path="/var/lib/kubelet/pods/bf70af16-418f-4fa5-a904-75988861ce47/volumes" Nov 04 01:20:53.350761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:53.350747 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf97e290-05ba-490e-834c-d34e195b76b3" path="/var/lib/kubelet/pods/cf97e290-05ba-490e-834c-d34e195b76b3/volumes" Nov 04 01:20:55.067315 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:55.067279 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2"] Nov 04 01:20:55.114643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:55.114610 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-044e8eab-r2mh2"] Nov 04 01:20:55.353648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:55.353558 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864b6246-557c-4e6d-8948-b8227df04e67" path="/var/lib/kubelet/pods/864b6246-557c-4e6d-8948-b8227df04e67/volumes" Nov 04 01:20:59.770382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.770339 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv"] Nov 04 01:20:59.771314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.771294 4486 cpu_manager.go:414] "containerMap: removing container" podUID="37aaabab-a61e-43a2-a92f-6b5a2517b1ab" containerName="revoke-keys" Nov 04 01:20:59.771417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.771315 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37aaabab-a61e-43a2-a92f-6b5a2517b1ab" containerName="revoke-keys" Nov 04 01:20:59.771474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.771467 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="37aaabab-a61e-43a2-a92f-6b5a2517b1ab" containerName="revoke-keys" Nov 04 01:20:59.777235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.777212 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.827064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.827037 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv"] Nov 04 01:20:59.844284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.844253 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.844471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.844375 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.844561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.844544 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpvw\" (UniqueName: \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.866958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.866931 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:59.867117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.867095 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:20:59.946095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.946071 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.946239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.946225 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.946362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.946348 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpvw\" (UniqueName: \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.946883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.946855 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:20:59.948404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:20:59.948383 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:21:00.001113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:00.001089 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpvw\" (UniqueName: \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\") pod \"maas-api-revoke-sub-fd39b8ef-5fzsv\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:21:00.091640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:00.091601 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:21:00.472650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:00.472623 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv"] Nov 04 01:21:01.261603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.261562 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" event={"ID":"0a79bb77-1b36-4917-bcc9-90b877a937a3","Type":"ContainerStarted","Data":"edf755f6e9ea9ac709ab39dc8ff2371b6b7b2b56259046201130d0a1e73dfc9a"} Nov 04 01:21:01.262016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.261631 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" event={"ID":"0a79bb77-1b36-4917-bcc9-90b877a937a3","Type":"ContainerStarted","Data":"b40500fef529492b22159a387b8692faf8f7f440f35732aa92b1fdc7140d48b9"} Nov 04 01:21:01.375355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.375299 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" podStartSLOduration=2.37528161 podStartE2EDuration="2.37528161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:01.370016783 +0000 UTC m=+1430.128678944" watchObservedRunningTime="2026-11-04 01:21:01.37528161 +0000 UTC m=+1430.133943773" Nov 04 01:21:01.453881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.453854 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc"] Nov 04 01:21:01.457941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.457926 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.510606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.510579 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc"] Nov 04 01:21:01.568252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.568217 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.568252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.568248 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77dwr\" (UniqueName: \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.568465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.568277 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.669378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.669348 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.669378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.669379 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77dwr\" (UniqueName: \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.669552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.669406 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.670143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.670124 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.671641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.671616 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.723636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.723614 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77dwr\" (UniqueName: \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\") pod \"maas-api-revoke-sub-20839ace-xvqjc\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:01.770738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:01.770715 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:02.142430 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:02.142399 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee56a7d_4a74_47c0_bba9_d4c3618ac499.slice/crio-054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746 WatchSource:0}: Error finding container 054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746: Status 404 returned error can't find the container with id 054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746 Nov 04 01:21:02.143901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:02.143871 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc"] Nov 04 01:21:02.271806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:02.271773 4486 generic.go:356] "Generic (PLEG): container finished" podID="0a79bb77-1b36-4917-bcc9-90b877a937a3" containerID="edf755f6e9ea9ac709ab39dc8ff2371b6b7b2b56259046201130d0a1e73dfc9a" exitCode=0 Nov 04 01:21:02.272203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:02.271820 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" event={"ID":"0a79bb77-1b36-4917-bcc9-90b877a937a3","Type":"ContainerDied","Data":"edf755f6e9ea9ac709ab39dc8ff2371b6b7b2b56259046201130d0a1e73dfc9a"} Nov 04 01:21:02.273270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:02.273251 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" event={"ID":"fee56a7d-4a74-47c0-bba9-d4c3618ac499","Type":"ContainerStarted","Data":"054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746"} Nov 04 01:21:03.050336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.050303 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb"] Nov 04 01:21:03.054675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.054642 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4202f8d0-tgglb"] Nov 04 01:21:03.284269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.284234 4486 generic.go:356] "Generic (PLEG): container finished" podID="fee56a7d-4a74-47c0-bba9-d4c3618ac499" containerID="9c634da9068349f67064eac132313ad9d62a09c45c4457b59cc032eb85b0b00d" exitCode=0 Nov 04 01:21:03.284723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.284325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" event={"ID":"fee56a7d-4a74-47c0-bba9-d4c3618ac499","Type":"ContainerDied","Data":"9c634da9068349f67064eac132313ad9d62a09c45c4457b59cc032eb85b0b00d"} Nov 04 01:21:03.352754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.350243 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5571fee-a1bd-414d-8365-ab602361e074" path="/var/lib/kubelet/pods/a5571fee-a1bd-414d-8365-ab602361e074/volumes" Nov 04 01:21:03.451507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.451482 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:21:03.595337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.595301 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\") pod \"0a79bb77-1b36-4917-bcc9-90b877a937a3\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " Nov 04 01:21:03.595515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.595349 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\") pod \"0a79bb77-1b36-4917-bcc9-90b877a937a3\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " Nov 04 01:21:03.595515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.595394 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\" (UniqueName: \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\") pod \"0a79bb77-1b36-4917-bcc9-90b877a937a3\" (UID: \"0a79bb77-1b36-4917-bcc9-90b877a937a3\") " Nov 04 01:21:03.595730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.595703 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca" pod "0a79bb77-1b36-4917-bcc9-90b877a937a3" (UID: "0a79bb77-1b36-4917-bcc9-90b877a937a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:03.595968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.595939 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:03.598028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.597945 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca" pod "0a79bb77-1b36-4917-bcc9-90b877a937a3" (UID: "0a79bb77-1b36-4917-bcc9-90b877a937a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:03.608237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.608208 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw" pod "0a79bb77-1b36-4917-bcc9-90b877a937a3" (UID: "0a79bb77-1b36-4917-bcc9-90b877a937a3"). InnerVolumeSpecName "kube-api-access-xvpvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:03.697155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.697132 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a79bb77-1b36-4917-bcc9-90b877a937a3-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:03.697278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:03.697159 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpvw\" (UniqueName: \"kubernetes.io/projected/0a79bb77-1b36-4917-bcc9-90b877a937a3-kube-api-access-xvpvw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:04.296188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.296153 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" Nov 04 01:21:04.296581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.296183 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv" event={"ID":"0a79bb77-1b36-4917-bcc9-90b877a937a3","Type":"ContainerDied","Data":"b40500fef529492b22159a387b8692faf8f7f440f35732aa92b1fdc7140d48b9"} Nov 04 01:21:04.296581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.296238 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40500fef529492b22159a387b8692faf8f7f440f35732aa92b1fdc7140d48b9" Nov 04 01:21:04.477565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.477532 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:04.618707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.618671 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\") pod \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " Nov 04 01:21:04.618904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.618827 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\" (UniqueName: \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\") pod \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " Nov 04 01:21:04.618904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.618861 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\") pod \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\" (UID: \"fee56a7d-4a74-47c0-bba9-d4c3618ac499\") " Nov 04 01:21:04.619334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.619306 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca" pod "fee56a7d-4a74-47c0-bba9-d4c3618ac499" (UID: "fee56a7d-4a74-47c0-bba9-d4c3618ac499"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:04.619567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.619545 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:04.621068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.621042 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca" pod "fee56a7d-4a74-47c0-bba9-d4c3618ac499" (UID: "fee56a7d-4a74-47c0-bba9-d4c3618ac499"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:04.621342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.621316 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr" pod "fee56a7d-4a74-47c0-bba9-d4c3618ac499" (UID: "fee56a7d-4a74-47c0-bba9-d4c3618ac499"). InnerVolumeSpecName "kube-api-access-77dwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:04.720934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.720907 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee56a7d-4a74-47c0-bba9-d4c3618ac499-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:04.720934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.720931 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77dwr\" (UniqueName: \"kubernetes.io/projected/fee56a7d-4a74-47c0-bba9-d4c3618ac499-kube-api-access-77dwr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:04.949492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.949421 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:21:04.950048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950034 4486 cpu_manager.go:414] "containerMap: removing container" podUID="fee56a7d-4a74-47c0-bba9-d4c3618ac499" containerName="revoke-keys" Nov 04 01:21:04.950106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950050 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee56a7d-4a74-47c0-bba9-d4c3618ac499" containerName="revoke-keys" Nov 04 01:21:04.950106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950080 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0a79bb77-1b36-4917-bcc9-90b877a937a3" containerName="revoke-keys" Nov 04 01:21:04.950106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950086 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a79bb77-1b36-4917-bcc9-90b877a937a3" containerName="revoke-keys" Nov 04 01:21:04.950198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950182 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee56a7d-4a74-47c0-bba9-d4c3618ac499" containerName="revoke-keys" Nov 04 01:21:04.950198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.950196 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a79bb77-1b36-4917-bcc9-90b877a937a3" containerName="revoke-keys" Nov 04 01:21:04.953712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:04.953692 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.058957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.058924 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:21:05.063182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.063154 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5"] Nov 04 01:21:05.066784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.066757 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0cd2e39-64rs5"] Nov 04 01:21:05.126551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126518 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126550 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126568 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126633 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126681 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pb6\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126779 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.126925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.126913 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.127046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.127034 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.127092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.127054 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228260 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228352 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228370 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228419 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228445 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228461 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228482 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228508 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pb6\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228578 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228871 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.228954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228911 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.229025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.228943 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.229237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.229211 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.229567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.229545 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.230817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.230795 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.231099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.231082 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.283063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.283036 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.283403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.283384 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pb6\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.304818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.304796 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" Nov 04 01:21:05.305229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.304825 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc" event={"ID":"fee56a7d-4a74-47c0-bba9-d4c3618ac499","Type":"ContainerDied","Data":"054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746"} Nov 04 01:21:05.305229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.304884 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054bcc63a632d6535ea55cc64fd748bd3b227a4a41e849a9973e29f9b7428746" Nov 04 01:21:05.350396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.350370 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3" path="/var/lib/kubelet/pods/59e7f85d-f9e0-4f3d-9bb1-0070d14d24b3/volumes" Nov 04 01:21:05.557324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.557245 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k"] Nov 04 01:21:05.562699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.562682 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.567898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.567869 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:05.612865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.612788 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:21:05.613050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.612973 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:05.621630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.621594 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k"] Nov 04 01:21:05.737788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.737756 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfddf\" (UniqueName: \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.738064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.738043 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.738114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.738090 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.839174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.839143 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.839358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.839205 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.839358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.839286 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfddf\" (UniqueName: \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.839919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.839898 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.841435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.841412 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.894238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.894216 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfddf\" (UniqueName: \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\") pod \"maas-api-revoke-sub-d0d3c9da-t8w4k\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:05.997150 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:05.997116 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c05fa5e_ecc6_4d08_8cec_b23ecc3b5e4c.slice/crio-851939da9128c3168dae0083eb63cb2cce087465ba1d5c1c47b3e6cbdcd024e7 WatchSource:0}: Error finding container 851939da9128c3168dae0083eb63cb2cce087465ba1d5c1c47b3e6cbdcd024e7: Status 404 returned error can't find the container with id 851939da9128c3168dae0083eb63cb2cce087465ba1d5c1c47b3e6cbdcd024e7 Nov 04 01:21:05.997962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.997934 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:21:05.999693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.999661 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:05.999777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.999734 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:05.999831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:05.999773 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:06.048287 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.048263 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv"] Nov 04 01:21:06.052751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.052726 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-128f4abf-v57nv"] Nov 04 01:21:06.180302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.180212 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:06.329605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.329567 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" event={"ID":"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c","Type":"ContainerStarted","Data":"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b"} Nov 04 01:21:06.340296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.329628 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" event={"ID":"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c","Type":"ContainerStarted","Data":"851939da9128c3168dae0083eb63cb2cce087465ba1d5c1c47b3e6cbdcd024e7"} Nov 04 01:21:06.444791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.444698 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podStartSLOduration=2.444683991 podStartE2EDuration="2.444683991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:06.441689923 +0000 UTC m=+1435.200352085" watchObservedRunningTime="2026-11-04 01:21:06.444683991 +0000 UTC m=+1435.203346151" Nov 04 01:21:06.561286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.558437 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k"] Nov 04 01:21:06.572584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:06.569314 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:07.373246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.373210 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4180686-7c78-4fbf-98cc-34a5507cf425" path="/var/lib/kubelet/pods/d4180686-7c78-4fbf-98cc-34a5507cf425/volumes" Nov 04 01:21:07.379369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.379335 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" event={"ID":"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda","Type":"ContainerStarted","Data":"0bbf578e064b206c44c84f3f23ce7cd02bc57f644162e1f367f1c43bdabf3148"} Nov 04 01:21:07.379532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.379401 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" event={"ID":"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda","Type":"ContainerStarted","Data":"a83d96c950c9b60f2b50f901af3041ffae9f2d19bebe6e56abb746082432f19a"} Nov 04 01:21:07.502798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.502746 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" podStartSLOduration=2.502730922 podStartE2EDuration="2.502730922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:07.499523256 +0000 UTC m=+1436.258185419" watchObservedRunningTime="2026-11-04 01:21:07.502730922 +0000 UTC m=+1436.261393083" Nov 04 01:21:07.570189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.569305 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.570189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:07.569364 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:08.389271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:08.389240 4486 generic.go:356] "Generic (PLEG): container finished" podID="2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" containerID="0bbf578e064b206c44c84f3f23ce7cd02bc57f644162e1f367f1c43bdabf3148" exitCode=0 Nov 04 01:21:08.389643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:08.389325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" event={"ID":"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda","Type":"ContainerDied","Data":"0bbf578e064b206c44c84f3f23ce7cd02bc57f644162e1f367f1c43bdabf3148"} Nov 04 01:21:08.568792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:08.568760 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.569016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:08.568833 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:09.568333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.568303 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.568741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.568359 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:09.596407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.596380 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:09.695414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.695327 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\") pod \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " Nov 04 01:21:09.695570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.695429 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\" (UniqueName: \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\") pod \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " Nov 04 01:21:09.695570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.695521 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\") pod \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\" (UID: \"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda\") " Nov 04 01:21:09.695889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.695857 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca" pod "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" (UID: "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:09.696332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.696301 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:09.697657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.697636 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf" pod "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" (UID: "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda"). InnerVolumeSpecName "kube-api-access-lfddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:09.698096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.698073 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca" pod "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" (UID: "2fa613ab-3dc3-404e-bacb-1c07a9fe8fda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:09.797615 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.797580 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:09.797615 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:09.797616 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfddf\" (UniqueName: \"kubernetes.io/projected/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda-kube-api-access-lfddf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:10.059060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.058257 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42"] Nov 04 01:21:10.061004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.060192 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" containerName="revoke-keys" Nov 04 01:21:10.061004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.060217 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" containerName="revoke-keys" Nov 04 01:21:10.061004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.060557 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" containerName="revoke-keys" Nov 04 01:21:10.072823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.072793 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.096105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.096072 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42"] Nov 04 01:21:10.104171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.104145 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.104349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.104323 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth2c\" (UniqueName: \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.104474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.104405 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.205581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.205543 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.205581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.205581 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth2c\" (UniqueName: \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.205790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.205620 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.206178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.206157 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.207967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.207944 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.265245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.265207 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth2c\" (UniqueName: \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\") pod \"maas-api-revoke-sub-ddfcbf04-kkv42\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.387778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.387737 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:10.407998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.407967 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" Nov 04 01:21:10.408128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.407966 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k" event={"ID":"2fa613ab-3dc3-404e-bacb-1c07a9fe8fda","Type":"ContainerDied","Data":"a83d96c950c9b60f2b50f901af3041ffae9f2d19bebe6e56abb746082432f19a"} Nov 04 01:21:10.408188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.408139 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83d96c950c9b60f2b50f901af3041ffae9f2d19bebe6e56abb746082432f19a" Nov 04 01:21:10.569287 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.569244 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.569829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.569316 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:10.792447 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:10.792412 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d9b722_2852_4c06_89c4_ee13c845d2a0.slice/crio-76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d WatchSource:0}: Error finding container 76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d: Status 404 returned error can't find the container with id 76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d Nov 04 01:21:10.793512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:10.793486 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42"] Nov 04 01:21:11.420954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.420922 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" event={"ID":"f2d9b722-2852-4c06-89c4-ee13c845d2a0","Type":"ContainerStarted","Data":"fc41be01778e51dd15e6406a798dbc1317fc63dabf75b908516ec08c74240465"} Nov 04 01:21:11.421153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.420974 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" event={"ID":"f2d9b722-2852-4c06-89c4-ee13c845d2a0","Type":"ContainerStarted","Data":"76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d"} Nov 04 01:21:11.545006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.536345 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" podStartSLOduration=2.536325817 podStartE2EDuration="2.536325817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:11.534320748 +0000 UTC m=+1440.292982909" watchObservedRunningTime="2026-11-04 01:21:11.536325817 +0000 UTC m=+1440.294987978" Nov 04 01:21:11.569007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.568672 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.569007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.568720 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:11.666430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.666390 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4"] Nov 04 01:21:11.672192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.672127 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.713052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.711515 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4"] Nov 04 01:21:11.727037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.726975 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.727165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.727125 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.727377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.727354 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbb9\" (UniqueName: \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.836192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.835580 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.836192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.835748 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbb9\" (UniqueName: \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.836192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.835848 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.837107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.836890 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.840264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.840198 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.890609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.890572 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbb9\" (UniqueName: \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\") pod \"maas-api-revoke-sub-d40faa50-hkvw4\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:11.936518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.936441 4486 scope.go:122] "RemoveContainer" containerID="fe680930a3cfef9b8eae1db0da8299f5f414626abcc1cfb8dd82e5804a3a0d54" Nov 04 01:21:11.951364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.951337 4486 scope.go:122] "RemoveContainer" containerID="e41af80bff13daf68ca7d0305207d1f23de6f9312f32e388da394c5cee507fe6" Nov 04 01:21:11.965032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.964977 4486 scope.go:122] "RemoveContainer" containerID="7a076ac8a455ccfd328009f98365222914b10e346b792521b0ef177e615e267e" Nov 04 01:21:11.979416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.979382 4486 scope.go:122] "RemoveContainer" containerID="17be05eaf29f8f1f7722e90a834228d436af7fbffe8e927e379d14773b679082" Nov 04 01:21:11.994292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.993969 4486 scope.go:122] "RemoveContainer" containerID="fb448a1c05413e44a91220d0738ad47dc1b55cf2665ff696aebf556b6c6e2c0f" Nov 04 01:21:11.995511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:11.995492 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:12.043022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.042796 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7"] Nov 04 01:21:12.049533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.049326 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.123297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.123110 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7"] Nov 04 01:21:12.141841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.141812 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.142140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.142116 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.142295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.142279 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shrr\" (UniqueName: \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.243872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.243798 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.243872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.243840 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.244115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.243890 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shrr\" (UniqueName: \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.244574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.244547 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.246553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.246530 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.301572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.301547 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shrr\" (UniqueName: \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\") pod \"maas-api-revoke-sub-cd61dd02-p87q7\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.394516 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:12.394486 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3fadf6_120b_412a_9127_e104075957f1.slice/crio-b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f WatchSource:0}: Error finding container b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f: Status 404 returned error can't find the container with id b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f Nov 04 01:21:12.395432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.395407 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4"] Nov 04 01:21:12.415904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.415876 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:12.432001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.431817 4486 generic.go:356] "Generic (PLEG): container finished" podID="f2d9b722-2852-4c06-89c4-ee13c845d2a0" containerID="fc41be01778e51dd15e6406a798dbc1317fc63dabf75b908516ec08c74240465" exitCode=0 Nov 04 01:21:12.432001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.431885 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" event={"ID":"f2d9b722-2852-4c06-89c4-ee13c845d2a0","Type":"ContainerDied","Data":"fc41be01778e51dd15e6406a798dbc1317fc63dabf75b908516ec08c74240465"} Nov 04 01:21:12.433965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.433936 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" event={"ID":"2a3fadf6-120b-412a-9127-e104075957f1","Type":"ContainerStarted","Data":"b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f"} Nov 04 01:21:12.569932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.569096 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 04 01:21:12.569932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.569168 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 04 01:21:12.802591 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:12.802551 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ab2709_b5c2_43c7_a362_8b826e670f9f.slice/crio-52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0 WatchSource:0}: Error finding container 52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0: Status 404 returned error can't find the container with id 52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0 Nov 04 01:21:12.803692 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:12.803662 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7"] Nov 04 01:21:13.447428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.447334 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" event={"ID":"2a3fadf6-120b-412a-9127-e104075957f1","Type":"ContainerStarted","Data":"2a31365e04a577fb285edc23cf2dac2c08118949bbc7407bfb40390f094f17e7"} Nov 04 01:21:13.449179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.449150 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" event={"ID":"00ab2709-b5c2-43c7-a362-8b826e670f9f","Type":"ContainerStarted","Data":"692d7e772b6babe5c65e188cd529331d5bd170b6b4b2401a7ce33f5e5d1c3c96"} Nov 04 01:21:13.449344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.449193 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" event={"ID":"00ab2709-b5c2-43c7-a362-8b826e670f9f","Type":"ContainerStarted","Data":"52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0"} Nov 04 01:21:13.561920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.561119 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" podStartSLOduration=2.561101256 podStartE2EDuration="2.561101256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:13.558264121 +0000 UTC m=+1442.316926282" watchObservedRunningTime="2026-11-04 01:21:13.561101256 +0000 UTC m=+1442.319763416" Nov 04 01:21:13.594090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.592240 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:13.594090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.593374 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:13.594090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.594011 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:21:13.630802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.630777 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:13.665101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.665070 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\") pod \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " Nov 04 01:21:13.665411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.665392 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\") pod \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " Nov 04 01:21:13.665411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.665397 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca" pod "f2d9b722-2852-4c06-89c4-ee13c845d2a0" (UID: "f2d9b722-2852-4c06-89c4-ee13c845d2a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:13.665539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.665427 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\" (UniqueName: \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\") pod \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\" (UID: \"f2d9b722-2852-4c06-89c4-ee13c845d2a0\") " Nov 04 01:21:13.666101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.666076 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:13.667717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.667689 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca" pod "f2d9b722-2852-4c06-89c4-ee13c845d2a0" (UID: "f2d9b722-2852-4c06-89c4-ee13c845d2a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:13.667801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.667783 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c" pod "f2d9b722-2852-4c06-89c4-ee13c845d2a0" (UID: "f2d9b722-2852-4c06-89c4-ee13c845d2a0"). InnerVolumeSpecName "kube-api-access-sth2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:13.671022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.670964 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" podStartSLOduration=2.6709515169999998 podStartE2EDuration="2.670951517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:13.668820298 +0000 UTC m=+1442.427482457" watchObservedRunningTime="2026-11-04 01:21:13.670951517 +0000 UTC m=+1442.429613676" Nov 04 01:21:13.767201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.767131 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2d9b722-2852-4c06-89c4-ee13c845d2a0-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:13.767201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:13.767157 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sth2c\" (UniqueName: \"kubernetes.io/projected/f2d9b722-2852-4c06-89c4-ee13c845d2a0-kube-api-access-sth2c\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:13.838563 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:21:13.838527 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:21:14.462904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.462616 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" Nov 04 01:21:14.462904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.462604 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42" event={"ID":"f2d9b722-2852-4c06-89c4-ee13c845d2a0","Type":"ContainerDied","Data":"76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d"} Nov 04 01:21:14.462904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.462716 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76139d8c168614217522bc26928acda4078cfecdafa1a462ec8ea80d9d95208d" Nov 04 01:21:14.464708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.464680 4486 generic.go:356] "Generic (PLEG): container finished" podID="2a3fadf6-120b-412a-9127-e104075957f1" containerID="2a31365e04a577fb285edc23cf2dac2c08118949bbc7407bfb40390f094f17e7" exitCode=0 Nov 04 01:21:14.464839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.464760 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" event={"ID":"2a3fadf6-120b-412a-9127-e104075957f1","Type":"ContainerDied","Data":"2a31365e04a577fb285edc23cf2dac2c08118949bbc7407bfb40390f094f17e7"} Nov 04 01:21:14.466554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.466530 4486 generic.go:356] "Generic (PLEG): container finished" podID="00ab2709-b5c2-43c7-a362-8b826e670f9f" containerID="692d7e772b6babe5c65e188cd529331d5bd170b6b4b2401a7ce33f5e5d1c3c96" exitCode=0 Nov 04 01:21:14.466633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:14.466605 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" event={"ID":"00ab2709-b5c2-43c7-a362-8b826e670f9f","Type":"ContainerDied","Data":"692d7e772b6babe5c65e188cd529331d5bd170b6b4b2401a7ce33f5e5d1c3c96"} Nov 04 01:21:15.648024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.647996 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:15.694492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.694461 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\" (UniqueName: \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\") pod \"2a3fadf6-120b-412a-9127-e104075957f1\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " Nov 04 01:21:15.694615 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.694573 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\") pod \"2a3fadf6-120b-412a-9127-e104075957f1\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " Nov 04 01:21:15.694696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.694659 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\") pod \"2a3fadf6-120b-412a-9127-e104075957f1\" (UID: \"2a3fadf6-120b-412a-9127-e104075957f1\") " Nov 04 01:21:15.695003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.694957 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca" pod "2a3fadf6-120b-412a-9127-e104075957f1" (UID: "2a3fadf6-120b-412a-9127-e104075957f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:15.695384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.695357 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:15.696665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.696641 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca" pod "2a3fadf6-120b-412a-9127-e104075957f1" (UID: "2a3fadf6-120b-412a-9127-e104075957f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:15.696786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.696765 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9" pod "2a3fadf6-120b-412a-9127-e104075957f1" (UID: "2a3fadf6-120b-412a-9127-e104075957f1"). InnerVolumeSpecName "kube-api-access-9xbb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:15.707640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.707621 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:15.796206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.796105 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\" (UniqueName: \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\") pod \"00ab2709-b5c2-43c7-a362-8b826e670f9f\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " Nov 04 01:21:15.796206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.796198 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\") pod \"00ab2709-b5c2-43c7-a362-8b826e670f9f\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " Nov 04 01:21:15.796435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.796244 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\") pod \"00ab2709-b5c2-43c7-a362-8b826e670f9f\" (UID: \"00ab2709-b5c2-43c7-a362-8b826e670f9f\") " Nov 04 01:21:15.796959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.796928 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca" pod "00ab2709-b5c2-43c7-a362-8b826e670f9f" (UID: "00ab2709-b5c2-43c7-a362-8b826e670f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:15.797087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.796956 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbb9\" (UniqueName: \"kubernetes.io/projected/2a3fadf6-120b-412a-9127-e104075957f1-kube-api-access-9xbb9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:15.797087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.797008 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a3fadf6-120b-412a-9127-e104075957f1-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:15.798340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.798315 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca" pod "00ab2709-b5c2-43c7-a362-8b826e670f9f" (UID: "00ab2709-b5c2-43c7-a362-8b826e670f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:15.798446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.798367 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr" pod "00ab2709-b5c2-43c7-a362-8b826e670f9f" (UID: "00ab2709-b5c2-43c7-a362-8b826e670f9f"). InnerVolumeSpecName "kube-api-access-6shrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:15.898219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.898186 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:15.898219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.898220 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00ab2709-b5c2-43c7-a362-8b826e670f9f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:15.898394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:15.898236 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shrr\" (UniqueName: \"kubernetes.io/projected/00ab2709-b5c2-43c7-a362-8b826e670f9f-kube-api-access-6shrr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:16.061731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.061641 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx"] Nov 04 01:21:16.068846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.068816 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-062c8bcb-qs4sx"] Nov 04 01:21:16.485194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.485165 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" Nov 04 01:21:16.485364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.485203 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7" event={"ID":"00ab2709-b5c2-43c7-a362-8b826e670f9f","Type":"ContainerDied","Data":"52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0"} Nov 04 01:21:16.485364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.485247 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52978c143e834e996e678d7496877677bb5da0248d980047764b0beea6f12ec0" Nov 04 01:21:16.486814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.486793 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" event={"ID":"2a3fadf6-120b-412a-9127-e104075957f1","Type":"ContainerDied","Data":"b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f"} Nov 04 01:21:16.486926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.486827 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b70dc780fa8f3fbdce6ed046f8a4869e4ebf5043ec14726a93255cbf708f481f" Nov 04 01:21:16.486926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:16.486801 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4" Nov 04 01:21:17.353700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:17.353661 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb" path="/var/lib/kubelet/pods/d21a68e4-9d4d-4e9c-acde-eaf1c06b88eb/volumes" Nov 04 01:21:18.055310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.055272 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8"] Nov 04 01:21:18.096279 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.096247 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc5c79d4-mbng8"] Nov 04 01:21:18.468862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.468830 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn"] Nov 04 01:21:18.469635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469617 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2a3fadf6-120b-412a-9127-e104075957f1" containerName="revoke-keys" Nov 04 01:21:18.469679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469636 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3fadf6-120b-412a-9127-e104075957f1" containerName="revoke-keys" Nov 04 01:21:18.469679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469656 4486 cpu_manager.go:414] "containerMap: removing container" podUID="00ab2709-b5c2-43c7-a362-8b826e670f9f" containerName="revoke-keys" Nov 04 01:21:18.469679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469662 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ab2709-b5c2-43c7-a362-8b826e670f9f" containerName="revoke-keys" Nov 04 01:21:18.469679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469673 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f2d9b722-2852-4c06-89c4-ee13c845d2a0" containerName="revoke-keys" Nov 04 01:21:18.469679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469679 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d9b722-2852-4c06-89c4-ee13c845d2a0" containerName="revoke-keys" Nov 04 01:21:18.469822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469771 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3fadf6-120b-412a-9127-e104075957f1" containerName="revoke-keys" Nov 04 01:21:18.469822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469780 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ab2709-b5c2-43c7-a362-8b826e670f9f" containerName="revoke-keys" Nov 04 01:21:18.469822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.469787 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d9b722-2852-4c06-89c4-ee13c845d2a0" containerName="revoke-keys" Nov 04 01:21:18.474652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.474633 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.527674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.527644 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn"] Nov 04 01:21:18.534605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.534578 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.534753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.534633 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8lz\" (UniqueName: \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.534814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.534788 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.559741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.559713 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:18.559853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.559778 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:21:18.635534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.635507 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.635687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.635668 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.635724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.635713 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8lz\" (UniqueName: \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.636284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.636260 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.637775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.637756 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.710007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.705644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8lz\" (UniqueName: \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\") pod \"maas-api-revoke-sub-5feeb212-hq2jn\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:18.790508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:18.790190 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:19.115863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.114229 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr"] Nov 04 01:21:19.126267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.121967 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49fd4126-gwfjr"] Nov 04 01:21:19.253014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.252651 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn"] Nov 04 01:21:19.333897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.333868 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:21:19.343009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.340663 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.369880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.369816 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29" path="/var/lib/kubelet/pods/9251a9d4-b817-40b4-ac4f-8ad4e3e6ec29/volumes" Nov 04 01:21:19.370766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.370742 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d252a4f3-c1aa-4dec-884f-7a9d46ffa849" path="/var/lib/kubelet/pods/d252a4f3-c1aa-4dec-884f-7a9d46ffa849/volumes" Nov 04 01:21:19.400317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.400277 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:21:19.426910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.426873 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.427702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.427676 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.427971 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428219 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428392 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chspz\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428496 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428610 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428714 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.428882 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.428869 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544287 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544372 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chspz\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544406 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544456 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544486 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544556 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.544737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544697 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.547391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544777 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.547391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.544824 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.547391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.546826 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.547809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.547509 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.548539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.548507 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.549421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.549390 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.550192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.550169 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.552774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.552701 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.554158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.554098 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.709483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.650168 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chspz\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:19.922185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.917970 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" event={"ID":"fe567b29-b506-4583-9587-b4e9f7fa21b7","Type":"ContainerStarted","Data":"f6bbaa451d805f4b01fd8efb573e64e34aad1ab72cd0e1f4233cd0748ff9c6bb"} Nov 04 01:21:19.987060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:19.983050 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:20.270058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:20.269237 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:21.150030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.132301 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.172007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.170301 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.209006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.172376 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.300066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.296847 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.407122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.399878 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:21.407122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.400004 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:21.966082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.966033 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:21.966270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.966131 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:21.966270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.966175 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:21.983013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:21.982960 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:21:22.125183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.125139 4486 generic.go:356] "Generic (PLEG): container finished" podID="fe567b29-b506-4583-9587-b4e9f7fa21b7" containerID="d2af75582248023decc03db37f577074636440453c43ecc7d25cbfb2c701d56b" exitCode=0 Nov 04 01:21:22.125380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.125209 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" event={"ID":"fe567b29-b506-4583-9587-b4e9f7fa21b7","Type":"ContainerDied","Data":"d2af75582248023decc03db37f577074636440453c43ecc7d25cbfb2c701d56b"} Nov 04 01:21:22.254141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.254096 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.288080 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:21:22.625011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.598608 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:22.633122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.633085 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:22.634169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:22.634147 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:21:23.278200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.278067 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:23.297018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.289165 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:21:23.298926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.298890 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:21:23.343152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.343114 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" event={"ID":"2ec10c8c-b173-4cd3-8c66-b4a135253bef","Type":"ContainerStarted","Data":"b07f3c45b17fc439bceae7930465deb0e6917a7519139369fbec1b6a2a3e1d14"} Nov 04 01:21:23.347006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.343384 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" event={"ID":"2ec10c8c-b173-4cd3-8c66-b4a135253bef","Type":"ContainerStarted","Data":"81b87964e3f7a36c5c1e0c0bf54e562d3879c747ba7ce64ca5c574ecdd77aa7f"} Nov 04 01:21:23.412521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.402455 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:21:23.486041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.485508 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podStartSLOduration=4.4854888 podStartE2EDuration="4.4854888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:23.470426622 +0000 UTC m=+1452.229088785" watchObservedRunningTime="2026-11-04 01:21:23.4854888 +0000 UTC m=+1452.244150959" Nov 04 01:21:23.554096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.545813 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:21:23.598545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.595972 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:21:23.810398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.810324 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:23.894717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.894687 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\") pod \"fe567b29-b506-4583-9587-b4e9f7fa21b7\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " Nov 04 01:21:23.894876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.894834 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\" (UniqueName: \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\") pod \"fe567b29-b506-4583-9587-b4e9f7fa21b7\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " Nov 04 01:21:23.895001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.894969 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\") pod \"fe567b29-b506-4583-9587-b4e9f7fa21b7\" (UID: \"fe567b29-b506-4583-9587-b4e9f7fa21b7\") " Nov 04 01:21:23.895691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.895128 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca" pod "fe567b29-b506-4583-9587-b4e9f7fa21b7" (UID: "fe567b29-b506-4583-9587-b4e9f7fa21b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:23.895691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.895588 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:23.897021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.896999 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca" pod "fe567b29-b506-4583-9587-b4e9f7fa21b7" (UID: "fe567b29-b506-4583-9587-b4e9f7fa21b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:23.897305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:23.897289 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz" pod "fe567b29-b506-4583-9587-b4e9f7fa21b7" (UID: "fe567b29-b506-4583-9587-b4e9f7fa21b7"). InnerVolumeSpecName "kube-api-access-fx8lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:24.000687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.000639 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe567b29-b506-4583-9587-b4e9f7fa21b7-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:24.000873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.000699 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8lz\" (UniqueName: \"kubernetes.io/projected/fe567b29-b506-4583-9587-b4e9f7fa21b7-kube-api-access-fx8lz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:24.273205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.272499 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:24.273479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.273452 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:24.273635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.273612 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:24.375004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.369217 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" event={"ID":"fe567b29-b506-4583-9587-b4e9f7fa21b7","Type":"ContainerDied","Data":"f6bbaa451d805f4b01fd8efb573e64e34aad1ab72cd0e1f4233cd0748ff9c6bb"} Nov 04 01:21:24.375004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.369266 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn" Nov 04 01:21:24.375004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.369311 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6bbaa451d805f4b01fd8efb573e64e34aad1ab72cd0e1f4233cd0748ff9c6bb" Nov 04 01:21:24.515838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.515807 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h"] Nov 04 01:21:24.516580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.516564 4486 cpu_manager.go:414] "containerMap: removing container" podUID="fe567b29-b506-4583-9587-b4e9f7fa21b7" containerName="revoke-keys" Nov 04 01:21:24.516633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.516581 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe567b29-b506-4583-9587-b4e9f7fa21b7" containerName="revoke-keys" Nov 04 01:21:24.516749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.516740 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe567b29-b506-4583-9587-b4e9f7fa21b7" containerName="revoke-keys" Nov 04 01:21:24.520445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.520427 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.571269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.571238 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:24.572181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.572155 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:21:24.577119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.577092 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h"] Nov 04 01:21:24.608965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.608939 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.609114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.609031 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6nds\" (UniqueName: \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.609196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.609173 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.710769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.710736 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.710951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.710816 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6nds\" (UniqueName: \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.710951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.710878 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.711463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.711439 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.713069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.713047 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.767995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.767947 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6nds\" (UniqueName: \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\") pod \"maas-api-revoke-sub-9c5c69b5-6nq6h\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:24.854006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:24.853211 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:25.255060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:25.251395 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h"] Nov 04 01:21:25.266908 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:25.266441 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b148509_0a4e_4173_b1f5_ac46ff4e7fa2.slice/crio-c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4 WatchSource:0}: Error finding container c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4: Status 404 returned error can't find the container with id c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4 Nov 04 01:21:25.270094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:25.270069 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:25.270202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:25.270114 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:25.382135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:25.382096 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" event={"ID":"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2","Type":"ContainerStarted","Data":"c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4"} Nov 04 01:21:26.184958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.184920 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:21:26.185214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.185191 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" containerID="cri-o://caf8826158a89a2c5b5f94fba5b56665faa6ccdffe04723c2c28c2e4978d6aa1" gracePeriod=30 Nov 04 01:21:26.271031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.266291 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:21:26.271031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.266537 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" containerID="cri-o://fd6cef264d96e8ceb5c06682ed8cb9e8ce7fdc132cce16f397d82668c6358eed" gracePeriod=30 Nov 04 01:21:26.271031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.270306 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:26.271031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.270352 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:26.273689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.273066 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:21:26.273689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.273268 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" containerID="cri-o://f54397dd35fb6ca1c3ca098033ae567347b5b97d2ca12b77cf04a450b9366837" gracePeriod=30 Nov 04 01:21:26.282857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.282226 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:21:26.282857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.282423 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" containerID="cri-o://daf9dafb7a24160376dd14f057004830e76e06c4beb1a814af45e312ace2dda1" gracePeriod=30 Nov 04 01:21:26.398557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.398519 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" event={"ID":"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2","Type":"ContainerStarted","Data":"6df1ebfcd25629642a79bd99a54c7b0f26f7af7a081ad466623db16de56a7e6c"} Nov 04 01:21:26.615360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.615328 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-q6c5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:21:26.615542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.615380 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:21:26.872644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:26.872535 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" podStartSLOduration=2.872520748 podStartE2EDuration="2.872520748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:26.870290921 +0000 UTC m=+1455.628953079" watchObservedRunningTime="2026-11-04 01:21:26.872520748 +0000 UTC m=+1455.631182908" Nov 04 01:21:27.275417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:27.275326 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:27.275417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:27.275376 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:27.407618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:27.407585 4486 generic.go:356] "Generic (PLEG): container finished" podID="6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" containerID="6df1ebfcd25629642a79bd99a54c7b0f26f7af7a081ad466623db16de56a7e6c" exitCode=0 Nov 04 01:21:27.407994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:27.407830 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" event={"ID":"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2","Type":"ContainerDied","Data":"6df1ebfcd25629642a79bd99a54c7b0f26f7af7a081ad466623db16de56a7e6c"} Nov 04 01:21:28.268180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.268137 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9"] Nov 04 01:21:28.271376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.271345 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:28.271535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.271389 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:28.273863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.273838 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.323611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.323585 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9"] Nov 04 01:21:28.411009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.410388 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4h8\" (UniqueName: \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.411009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.410530 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.411009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.410686 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.512136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.512076 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4h8\" (UniqueName: \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.512323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.512239 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.512430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.512406 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.512892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.512867 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.516008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.515405 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.587005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.585289 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4h8\" (UniqueName: \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\") pod \"maas-api-revoke-sub-b8b609b5-m25b9\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.588886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.588671 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:28.697253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.697225 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:28.822691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.822659 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\") pod \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " Nov 04 01:21:28.823341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.823317 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\" (UniqueName: \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\") pod \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " Nov 04 01:21:28.823901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.823248 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca" pod "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" (UID: "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:28.824077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.824046 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\") pod \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\" (UID: \"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2\") " Nov 04 01:21:28.824857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.824835 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:28.827041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.826494 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds" pod "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" (UID: "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2"). InnerVolumeSpecName "kube-api-access-t6nds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:28.869511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.869476 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca" pod "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" (UID: "6b148509-0a4e-4173-b1f5-ac46ff4e7fa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:28.925930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.925902 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6nds\" (UniqueName: \"kubernetes.io/projected/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-kube-api-access-t6nds\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:28.925930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:28.925928 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:29.068839 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:29.068573 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72014b47_7137_4ac4_a3f6_8ce3dd8901cd.slice/crio-89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb WatchSource:0}: Error finding container 89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb: Status 404 returned error can't find the container with id 89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb Nov 04 01:21:29.073243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.073180 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9"] Nov 04 01:21:29.280728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.272354 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:29.280728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.272489 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:29.288162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.288130 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc"] Nov 04 01:21:29.289183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.289131 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" containerName="revoke-keys" Nov 04 01:21:29.289183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.289152 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" containerName="revoke-keys" Nov 04 01:21:29.289357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.289256 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" containerName="revoke-keys" Nov 04 01:21:29.296553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.296534 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.380559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.371879 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.380559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.371941 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.380559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.372010 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf65h\" (UniqueName: \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.392867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.392651 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc"] Nov 04 01:21:29.475562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.475523 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.476123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.476099 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.476295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.476280 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf65h\" (UniqueName: \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.477482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.477246 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.477482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.477400 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" Nov 04 01:21:29.477607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.477547 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h" event={"ID":"6b148509-0a4e-4173-b1f5-ac46ff4e7fa2","Type":"ContainerDied","Data":"c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4"} Nov 04 01:21:29.477664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.477630 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04644e9492a91b1f610c2a07937b9bbd31b668b4b26b95966f865d8b4a1d4e4" Nov 04 01:21:29.480949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.480924 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.481251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.481157 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" event={"ID":"72014b47-7137-4ac4-a3f6-8ce3dd8901cd","Type":"ContainerStarted","Data":"89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb"} Nov 04 01:21:29.537368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.537334 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf65h\" (UniqueName: \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\") pod \"maas-api-revoke-sub-70447975-z8sdc\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.571247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.571212 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w"] Nov 04 01:21:29.575863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.575832 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.626871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.626837 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:29.639004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.638441 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w"] Nov 04 01:21:29.683245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.681110 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.683245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.681214 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.683245 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.681580 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpdl\" (UniqueName: \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.782796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.782760 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.782972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.782820 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.783090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.783067 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpdl\" (UniqueName: \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.783591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.783563 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.786653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.786625 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.840239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.840203 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpdl\" (UniqueName: \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\") pod \"maas-api-revoke-sub-7faf423c-hr72w\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:29.895735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:29.895653 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:30.066266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.066201 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc"] Nov 04 01:21:30.270825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.270789 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 04 01:21:30.271067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.270846 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 04 01:21:30.303259 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:30.303224 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386a2927_c251_44aa_9952_003291c700c8.slice/crio-ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663 WatchSource:0}: Error finding container ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663: Status 404 returned error can't find the container with id ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663 Nov 04 01:21:30.304316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.304290 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w"] Nov 04 01:21:30.339778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.339733 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t"] Nov 04 01:21:30.346779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.346756 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.397090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.395707 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.397090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.396010 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.397090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.396065 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zg2\" (UniqueName: \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.397090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.396469 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t"] Nov 04 01:21:30.495109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.495065 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" event={"ID":"94b17593-8487-4789-8923-48be2daada79","Type":"ContainerStarted","Data":"c712054746832bf74398be6e8ec2cf4228f23faea4eb0f5144899c4f019a1feb"} Nov 04 01:21:30.497290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.497259 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.497515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.497500 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.497612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.497547 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zg2\" (UniqueName: \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.498152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.498122 4486 generic.go:356] "Generic (PLEG): container finished" podID="72014b47-7137-4ac4-a3f6-8ce3dd8901cd" containerID="c235b9f95fa8d4350c339f4a7797a5dd9d6ff0beab605fe0669f4f8c9c0c9828" exitCode=0 Nov 04 01:21:30.498292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.498256 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" event={"ID":"72014b47-7137-4ac4-a3f6-8ce3dd8901cd","Type":"ContainerDied","Data":"c235b9f95fa8d4350c339f4a7797a5dd9d6ff0beab605fe0669f4f8c9c0c9828"} Nov 04 01:21:30.498292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.498281 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.499878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.499853 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" event={"ID":"386a2927-c251-44aa-9952-003291c700c8","Type":"ContainerStarted","Data":"ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663"} Nov 04 01:21:30.529896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.529710 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.553457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.553395 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zg2\" (UniqueName: \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\") pod \"maas-api-revoke-sub-a6a909cc-gjt8t\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:30.662265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:30.662224 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:31.069740 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:31.069708 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec1488c_9c35_4489_8d91_a774b21c1218.slice/crio-9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d WatchSource:0}: Error finding container 9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d: Status 404 returned error can't find the container with id 9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d Nov 04 01:21:31.071749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.071716 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t"] Nov 04 01:21:31.274643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.274570 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:31.275209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.275180 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:31.275907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.275889 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:21:31.585062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.584949 4486 generic.go:356] "Generic (PLEG): container finished" podID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerID="caf8826158a89a2c5b5f94fba5b56665faa6ccdffe04723c2c28c2e4978d6aa1" exitCode=0 Nov 04 01:21:31.587062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.585583 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerDied","Data":"caf8826158a89a2c5b5f94fba5b56665faa6ccdffe04723c2c28c2e4978d6aa1"} Nov 04 01:21:31.587280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.587266 4486 scope.go:122] "RemoveContainer" containerID="d7406a3eb9a46d23c655a0b9e1bab43429ed5d377d16b933795ff3b86104ee1b" Nov 04 01:21:31.591142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.591122 4486 generic.go:356] "Generic (PLEG): container finished" podID="386a2927-c251-44aa-9952-003291c700c8" containerID="34d4c7550d2ee34fdc9e9127a479bab59540f8b0976db29a4c8927f8b21cd85c" exitCode=0 Nov 04 01:21:31.591428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.591406 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" event={"ID":"386a2927-c251-44aa-9952-003291c700c8","Type":"ContainerDied","Data":"34d4c7550d2ee34fdc9e9127a479bab59540f8b0976db29a4c8927f8b21cd85c"} Nov 04 01:21:31.595176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.595150 4486 generic.go:356] "Generic (PLEG): container finished" podID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerID="f54397dd35fb6ca1c3ca098033ae567347b5b97d2ca12b77cf04a450b9366837" exitCode=0 Nov 04 01:21:31.595349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.595329 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" event={"ID":"409db713-1d4e-4c3d-8f69-8c402a4f52ec","Type":"ContainerDied","Data":"f54397dd35fb6ca1c3ca098033ae567347b5b97d2ca12b77cf04a450b9366837"} Nov 04 01:21:31.615099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.614930 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" event={"ID":"2ec1488c-9c35-4489-8d91-a774b21c1218","Type":"ContainerStarted","Data":"9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d"} Nov 04 01:21:31.635046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.634656 4486 generic.go:356] "Generic (PLEG): container finished" podID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerID="daf9dafb7a24160376dd14f057004830e76e06c4beb1a814af45e312ace2dda1" exitCode=0 Nov 04 01:21:31.635046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.634801 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" event={"ID":"9fd91c68-fd7d-4b3c-95aa-55680be62f4d","Type":"ContainerDied","Data":"daf9dafb7a24160376dd14f057004830e76e06c4beb1a814af45e312ace2dda1"} Nov 04 01:21:31.642578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.642550 4486 generic.go:356] "Generic (PLEG): container finished" podID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerID="fd6cef264d96e8ceb5c06682ed8cb9e8ce7fdc132cce16f397d82668c6358eed" exitCode=0 Nov 04 01:21:31.642673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.642648 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" event={"ID":"c4087ecc-3046-458d-a48a-9bce2b8ace63","Type":"ContainerDied","Data":"fd6cef264d96e8ceb5c06682ed8cb9e8ce7fdc132cce16f397d82668c6358eed"} Nov 04 01:21:31.643244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.643226 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:21:31.644466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.644439 4486 generic.go:356] "Generic (PLEG): container finished" podID="94b17593-8487-4789-8923-48be2daada79" containerID="66e84e4ac71d12f7e50c61a2ce9471c99780db9b52c21b4960fcf9a7ec8f54d6" exitCode=0 Nov 04 01:21:31.644560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.644516 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" event={"ID":"94b17593-8487-4789-8923-48be2daada79","Type":"ContainerDied","Data":"66e84e4ac71d12f7e50c61a2ce9471c99780db9b52c21b4960fcf9a7ec8f54d6"} Nov 04 01:21:31.702332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.702243 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:21:31.756588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756547 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.756725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756612 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.756725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756658 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.756858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756724 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.756858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756768 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.756858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756841 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.756943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756912 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.756995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.756946 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.757054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757034 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.757107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757091 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.757155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757129 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.757204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757180 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.757255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757232 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.757340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757320 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.757443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757424 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.757519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757462 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\") pod \"16c18ddb-ccc6-403d-b712-ba202a506ea3\" (UID: \"16c18ddb-ccc6-403d-b712-ba202a506ea3\") " Nov 04 01:21:31.757519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757512 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.757634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.757559 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\") pod \"c4087ecc-3046-458d-a48a-9bce2b8ace63\" (UID: \"c4087ecc-3046-458d-a48a-9bce2b8ace63\") " Nov 04 01:21:31.760738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.760699 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.761384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.761328 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.761583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.761560 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.762707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.762433 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.763283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.763032 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:31.764555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.764383 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.764972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.764761 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:31.766610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.766577 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.767857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.767834 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.768525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.768505 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.782503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.782405 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:21:31.784409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.784351 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.803548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.803504 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:21:31.804519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.803721 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.807024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.806873 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "kube-api-access-52zb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.812150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.807321 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy" pod "16c18ddb-ccc6-403d-b712-ba202a506ea3" (UID: "16c18ddb-ccc6-403d-b712-ba202a506ea3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.812150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.808153 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "kube-api-access-2rm4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.812150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.808444 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:21:31.812150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.809220 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy" pod "c4087ecc-3046-458d-a48a-9bce2b8ace63" (UID: "c4087ecc-3046-458d-a48a-9bce2b8ace63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.832415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.832380 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860216 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860246 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860256 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860266 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860276 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860285 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860293 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860301 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860311 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4087ecc-3046-458d-a48a-9bce2b8ace63-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860319 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860337 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860348 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zb8\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-kube-api-access-52zb8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860357 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860366 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860375 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c18ddb-ccc6-403d-b712-ba202a506ea3-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860383 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm4z\" (UniqueName: \"kubernetes.io/projected/c4087ecc-3046-458d-a48a-9bce2b8ace63-kube-api-access-2rm4z\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860392 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4087ecc-3046-458d-a48a-9bce2b8ace63-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.860881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.860400 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c18ddb-ccc6-403d-b712-ba202a506ea3-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.914248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.914224 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:31.960804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960774 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.960958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960818 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.960958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960859 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.960958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960879 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.960958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960912 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.960958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960944 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.960976 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.961257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961049 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961078 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961116 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961200 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961288 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961304 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961305 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961394 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961438 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961526 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.961618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961571 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\") pod \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\" (UID: \"9fd91c68-fd7d-4b3c-95aa-55680be62f4d\") " Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961704 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961770 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961802 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\") pod \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\" (UID: \"409db713-1d4e-4c3d-8f69-8c402a4f52ec\") " Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961832 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.961935 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962038 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:31.962347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962146 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962336 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962355 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962148 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.962617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962588 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962619 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962634 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962648 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962659 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962668 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962676 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962684 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962693 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.962754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.962702 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:31.965075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.965048 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:21:31.965183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.965074 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.965309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.965284 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.965383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.965358 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "kube-api-access-pxrjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.965825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.965797 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:31.966052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.966026 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:21:31.966657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.966614 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy" pod "9fd91c68-fd7d-4b3c-95aa-55680be62f4d" (UID: "9fd91c68-fd7d-4b3c-95aa-55680be62f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:21:31.966742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:31.966672 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x" pod "409db713-1d4e-4c3d-8f69-8c402a4f52ec" (UID: "409db713-1d4e-4c3d-8f69-8c402a4f52ec"). InnerVolumeSpecName "kube-api-access-cj92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.063921 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\" (UniqueName: \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\") pod \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064010 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\") pod \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064100 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\") pod \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\" (UID: \"72014b47-7137-4ac4-a3f6-8ce3dd8901cd\") " Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064768 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064787 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrjs\" (UniqueName: \"kubernetes.io/projected/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-kube-api-access-pxrjs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064802 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj92x\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-kube-api-access-cj92x\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064816 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064829 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064842 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd91c68-fd7d-4b3c-95aa-55680be62f4d-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064857 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.064870 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/409db713-1d4e-4c3d-8f69-8c402a4f52ec-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.065702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.065229 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca" pod "72014b47-7137-4ac4-a3f6-8ce3dd8901cd" (UID: "72014b47-7137-4ac4-a3f6-8ce3dd8901cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:32.067803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.067774 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8" pod "72014b47-7137-4ac4-a3f6-8ce3dd8901cd" (UID: "72014b47-7137-4ac4-a3f6-8ce3dd8901cd"). InnerVolumeSpecName "kube-api-access-kw4h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:32.068551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.068533 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca" pod "72014b47-7137-4ac4-a3f6-8ce3dd8901cd" (UID: "72014b47-7137-4ac4-a3f6-8ce3dd8901cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:32.159854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.159762 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" podStartSLOduration=2.159749545 podStartE2EDuration="2.159749545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:32.153676402 +0000 UTC m=+1460.912338566" watchObservedRunningTime="2026-11-04 01:21:32.159749545 +0000 UTC m=+1460.918411706" Nov 04 01:21:32.165828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.165797 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.165828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.165825 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4h8\" (UniqueName: \"kubernetes.io/projected/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-kube-api-access-kw4h8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.165965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.165835 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72014b47-7137-4ac4-a3f6-8ce3dd8901cd-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.433857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.433767 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c"] Nov 04 01:21:32.434495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434460 4486 cpu_manager.go:414] "containerMap: removing container" podUID="72014b47-7137-4ac4-a3f6-8ce3dd8901cd" containerName="revoke-keys" Nov 04 01:21:32.434495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434494 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72014b47-7137-4ac4-a3f6-8ce3dd8901cd" containerName="revoke-keys" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434534 4486 cpu_manager.go:414] "containerMap: removing container" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434542 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434554 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434562 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434584 4486 cpu_manager.go:414] "containerMap: removing container" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434591 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434611 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" Nov 04 01:21:32.434655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434619 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434789 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" containerName="istio-proxy" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434809 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434820 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="72014b47-7137-4ac4-a3f6-8ce3dd8901cd" containerName="revoke-keys" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434836 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434845 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" containerName="istio-proxy" Nov 04 01:21:32.435081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.434859 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" containerName="istio-proxy" Nov 04 01:21:32.438646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.438625 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.501438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.501382 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c"] Nov 04 01:21:32.571573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.571542 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5gv\" (UniqueName: \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.571717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.571610 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.571789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.571768 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.656770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.656739 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" Nov 04 01:21:32.657296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.656790 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz" event={"ID":"c4087ecc-3046-458d-a48a-9bce2b8ace63","Type":"ContainerDied","Data":"4744dd3ec05cb76c28a93d151c0f8e57461976b7d3b5d0d9bc71bd93ec2dde61"} Nov 04 01:21:32.657296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.656889 4486 scope.go:122] "RemoveContainer" containerID="fd6cef264d96e8ceb5c06682ed8cb9e8ce7fdc132cce16f397d82668c6358eed" Nov 04 01:21:32.659224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.659201 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" event={"ID":"72014b47-7137-4ac4-a3f6-8ce3dd8901cd","Type":"ContainerDied","Data":"89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb"} Nov 04 01:21:32.659345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.659249 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e5527882515417afe9fa5bf0aa1ee069ef5a23a72d16188d17b96a0b6628fb" Nov 04 01:21:32.659345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.659311 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9" Nov 04 01:21:32.667885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.667623 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" event={"ID":"16c18ddb-ccc6-403d-b712-ba202a506ea3","Type":"ContainerDied","Data":"2a578209bae5ae0eabbf2a784e2073962bfc7373146831e17e9cdd9b9c6bae05"} Nov 04 01:21:32.667885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.667701 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l" Nov 04 01:21:32.671071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.671037 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" Nov 04 01:21:32.671167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.671085 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs" event={"ID":"409db713-1d4e-4c3d-8f69-8c402a4f52ec","Type":"ContainerDied","Data":"0075bd35496714222e9695bfe2daf9c67eef8ed75d7347e8455bfb59b3a4e5e6"} Nov 04 01:21:32.675030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.672890 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5gv\" (UniqueName: \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.675030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.672950 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.675030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.673096 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.675030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.674309 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.677342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.677321 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.677867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.677842 4486 generic.go:356] "Generic (PLEG): container finished" podID="2ec1488c-9c35-4489-8d91-a774b21c1218" containerID="7934ad14399e67e30a02a6b99f87fab87cf1aa52c6618b0d9c2ff43ad4abc966" exitCode=0 Nov 04 01:21:32.677960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.677944 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" event={"ID":"2ec1488c-9c35-4489-8d91-a774b21c1218","Type":"ContainerDied","Data":"7934ad14399e67e30a02a6b99f87fab87cf1aa52c6618b0d9c2ff43ad4abc966"} Nov 04 01:21:32.679687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.679665 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" Nov 04 01:21:32.682106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.681460 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9" event={"ID":"9fd91c68-fd7d-4b3c-95aa-55680be62f4d","Type":"ContainerDied","Data":"d5082362b197c656ab8a803455def588458f1b189cb598648c47579dd4183003"} Nov 04 01:21:32.684100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.683853 4486 scope.go:122] "RemoveContainer" containerID="caf8826158a89a2c5b5f94fba5b56665faa6ccdffe04723c2c28c2e4978d6aa1" Nov 04 01:21:32.700112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.700090 4486 scope.go:122] "RemoveContainer" containerID="f54397dd35fb6ca1c3ca098033ae567347b5b97d2ca12b77cf04a450b9366837" Nov 04 01:21:32.715706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.715577 4486 scope.go:122] "RemoveContainer" containerID="daf9dafb7a24160376dd14f057004830e76e06c4beb1a814af45e312ace2dda1" Nov 04 01:21:32.731756 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.731727 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5gv\" (UniqueName: \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\") pod \"maas-api-revoke-sub-8517a231-6wq7c\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.753299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.753191 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:32.862601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.862521 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:32.930166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.930134 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:32.986740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.986664 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:21:32.992321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.992290 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-9tzv9"] Nov 04 01:21:32.996661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.996637 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\") pod \"94b17593-8487-4789-8923-48be2daada79\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " Nov 04 01:21:32.996839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.996822 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\" (UniqueName: \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\") pod \"94b17593-8487-4789-8923-48be2daada79\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " Nov 04 01:21:32.996974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.996958 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\") pod \"94b17593-8487-4789-8923-48be2daada79\" (UID: \"94b17593-8487-4789-8923-48be2daada79\") " Nov 04 01:21:32.997352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.997322 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca" pod "94b17593-8487-4789-8923-48be2daada79" (UID: "94b17593-8487-4789-8923-48be2daada79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:32.997578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.997558 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b17593-8487-4789-8923-48be2daada79-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:32.998783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.998756 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca" pod "94b17593-8487-4789-8923-48be2daada79" (UID: "94b17593-8487-4789-8923-48be2daada79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:32.998969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:32.998952 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h" pod "94b17593-8487-4789-8923-48be2daada79" (UID: "94b17593-8487-4789-8923-48be2daada79"). InnerVolumeSpecName "kube-api-access-zf65h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:33.098310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.098275 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\" (UniqueName: \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\") pod \"386a2927-c251-44aa-9952-003291c700c8\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " Nov 04 01:21:33.098509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.098317 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\") pod \"386a2927-c251-44aa-9952-003291c700c8\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " Nov 04 01:21:33.098509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.098480 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\") pod \"386a2927-c251-44aa-9952-003291c700c8\" (UID: \"386a2927-c251-44aa-9952-003291c700c8\") " Nov 04 01:21:33.098865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.098835 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca" pod "386a2927-c251-44aa-9952-003291c700c8" (UID: "386a2927-c251-44aa-9952-003291c700c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:33.099114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.099093 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/386a2927-c251-44aa-9952-003291c700c8-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:33.099209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.099118 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b17593-8487-4789-8923-48be2daada79-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:33.099209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.099133 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf65h\" (UniqueName: \"kubernetes.io/projected/94b17593-8487-4789-8923-48be2daada79-kube-api-access-zf65h\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:33.099335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.099253 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:21:33.100517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.100477 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca" pod "386a2927-c251-44aa-9952-003291c700c8" (UID: "386a2927-c251-44aa-9952-003291c700c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:33.100625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.100555 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl" pod "386a2927-c251-44aa-9952-003291c700c8" (UID: "386a2927-c251-44aa-9952-003291c700c8"). InnerVolumeSpecName "kube-api-access-jwpdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:33.104275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.104255 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-l62hz"] Nov 04 01:21:33.148560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.148527 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c"] Nov 04 01:21:33.149172 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:33.149146 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3151800_d67b_4e59_93fd_ec9eb5e23f74.slice/crio-c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066 WatchSource:0}: Error finding container c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066: Status 404 returned error can't find the container with id c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066 Nov 04 01:21:33.200517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.200493 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpdl\" (UniqueName: \"kubernetes.io/projected/386a2927-c251-44aa-9952-003291c700c8-kube-api-access-jwpdl\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:33.200603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.200522 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/386a2927-c251-44aa-9952-003291c700c8-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:33.207052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.207024 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:21:33.217453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.213160 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-5hlrs"] Nov 04 01:21:33.317178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.317148 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:21:33.322066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.322039 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-q6c5l"] Nov 04 01:21:33.351199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.351168 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" path="/var/lib/kubelet/pods/16c18ddb-ccc6-403d-b712-ba202a506ea3/volumes" Nov 04 01:21:33.351687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.351673 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409db713-1d4e-4c3d-8f69-8c402a4f52ec" path="/var/lib/kubelet/pods/409db713-1d4e-4c3d-8f69-8c402a4f52ec/volumes" Nov 04 01:21:33.352134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.352120 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd91c68-fd7d-4b3c-95aa-55680be62f4d" path="/var/lib/kubelet/pods/9fd91c68-fd7d-4b3c-95aa-55680be62f4d/volumes" Nov 04 01:21:33.352903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.352887 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4087ecc-3046-458d-a48a-9bce2b8ace63" path="/var/lib/kubelet/pods/c4087ecc-3046-458d-a48a-9bce2b8ace63/volumes" Nov 04 01:21:33.691309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.691207 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" event={"ID":"d3151800-d67b-4e59-93fd-ec9eb5e23f74","Type":"ContainerStarted","Data":"612ef46ba81c15f3f137e3b0e39b2e7ad8c5120a1b2586e644d04988b7e0f1d6"} Nov 04 01:21:33.691309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.691274 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" event={"ID":"d3151800-d67b-4e59-93fd-ec9eb5e23f74","Type":"ContainerStarted","Data":"c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066"} Nov 04 01:21:33.693831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.693804 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" Nov 04 01:21:33.693975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.693833 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc" event={"ID":"94b17593-8487-4789-8923-48be2daada79","Type":"ContainerDied","Data":"c712054746832bf74398be6e8ec2cf4228f23faea4eb0f5144899c4f019a1feb"} Nov 04 01:21:33.693975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.693876 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c712054746832bf74398be6e8ec2cf4228f23faea4eb0f5144899c4f019a1feb" Nov 04 01:21:33.696596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.696575 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" Nov 04 01:21:33.696721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.696609 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w" event={"ID":"386a2927-c251-44aa-9952-003291c700c8","Type":"ContainerDied","Data":"ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663"} Nov 04 01:21:33.696721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.696658 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca29a9d76f655aa725d449d8d74856c16c9873e96be21df3be2ed1637d923663" Nov 04 01:21:33.801844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.801775 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" podStartSLOduration=1.801754952 podStartE2EDuration="1.801754952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:33.800329895 +0000 UTC m=+1462.558992056" watchObservedRunningTime="2026-11-04 01:21:33.801754952 +0000 UTC m=+1462.560417114" Nov 04 01:21:33.867369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:33.867339 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:34.012208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.012134 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\") pod \"2ec1488c-9c35-4489-8d91-a774b21c1218\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " Nov 04 01:21:34.012208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.012193 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\") pod \"2ec1488c-9c35-4489-8d91-a774b21c1218\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " Nov 04 01:21:34.012399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.012383 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\" (UniqueName: \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\") pod \"2ec1488c-9c35-4489-8d91-a774b21c1218\" (UID: \"2ec1488c-9c35-4489-8d91-a774b21c1218\") " Nov 04 01:21:34.012577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.012553 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca" pod "2ec1488c-9c35-4489-8d91-a774b21c1218" (UID: "2ec1488c-9c35-4489-8d91-a774b21c1218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:34.012947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.012917 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:34.014560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.014513 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca" pod "2ec1488c-9c35-4489-8d91-a774b21c1218" (UID: "2ec1488c-9c35-4489-8d91-a774b21c1218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:34.015351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.014864 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2" pod "2ec1488c-9c35-4489-8d91-a774b21c1218" (UID: "2ec1488c-9c35-4489-8d91-a774b21c1218"). InnerVolumeSpecName "kube-api-access-v4zg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:34.049585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.049561 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx"] Nov 04 01:21:34.054296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.054273 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45bddcc9-nptnx"] Nov 04 01:21:34.114110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.114081 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec1488c-9c35-4489-8d91-a774b21c1218-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:34.114110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.114109 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4zg2\" (UniqueName: \"kubernetes.io/projected/2ec1488c-9c35-4489-8d91-a774b21c1218-kube-api-access-v4zg2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:34.443416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.443384 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:21:34.444040 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444025 4486 cpu_manager.go:414] "containerMap: removing container" podUID="94b17593-8487-4789-8923-48be2daada79" containerName="revoke-keys" Nov 04 01:21:34.444104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444041 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b17593-8487-4789-8923-48be2daada79" containerName="revoke-keys" Nov 04 01:21:34.444104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444092 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2ec1488c-9c35-4489-8d91-a774b21c1218" containerName="revoke-keys" Nov 04 01:21:34.444104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444099 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec1488c-9c35-4489-8d91-a774b21c1218" containerName="revoke-keys" Nov 04 01:21:34.444189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444107 4486 cpu_manager.go:414] "containerMap: removing container" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:34.444189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444112 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c18ddb-ccc6-403d-b712-ba202a506ea3" containerName="istio-proxy" Nov 04 01:21:34.444189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444123 4486 cpu_manager.go:414] "containerMap: removing container" podUID="386a2927-c251-44aa-9952-003291c700c8" containerName="revoke-keys" Nov 04 01:21:34.444189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444127 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386a2927-c251-44aa-9952-003291c700c8" containerName="revoke-keys" Nov 04 01:21:34.444298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444220 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec1488c-9c35-4489-8d91-a774b21c1218" containerName="revoke-keys" Nov 04 01:21:34.444298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444230 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="386a2927-c251-44aa-9952-003291c700c8" containerName="revoke-keys" Nov 04 01:21:34.444298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.444237 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b17593-8487-4789-8923-48be2daada79" containerName="revoke-keys" Nov 04 01:21:34.447843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.447824 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.504490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.504457 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:21:34.622298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622261 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622305 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622380 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622503 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622546 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfbx\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622659 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622728 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622753 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.622801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.622776 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.708722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.708637 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" Nov 04 01:21:34.708722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.708659 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t" event={"ID":"2ec1488c-9c35-4489-8d91-a774b21c1218","Type":"ContainerDied","Data":"9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d"} Nov 04 01:21:34.708722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.708716 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a4a6cc405a1b7b359254a5ff7bc548a6a8f2e00c0200b824ca4c42920676d0d" Nov 04 01:21:34.710783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.710755 4486 generic.go:356] "Generic (PLEG): container finished" podID="d3151800-d67b-4e59-93fd-ec9eb5e23f74" containerID="612ef46ba81c15f3f137e3b0e39b2e7ad8c5120a1b2586e644d04988b7e0f1d6" exitCode=0 Nov 04 01:21:34.710898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.710853 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" event={"ID":"d3151800-d67b-4e59-93fd-ec9eb5e23f74","Type":"ContainerDied","Data":"612ef46ba81c15f3f137e3b0e39b2e7ad8c5120a1b2586e644d04988b7e0f1d6"} Nov 04 01:21:34.723711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723688 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.723825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723728 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.723825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723751 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.723825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723818 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.723948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723852 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfbx\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.723948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723916 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.723976 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724034 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724079 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724150 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724277 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724447 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724466 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.724568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.724550 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.726209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.726182 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.726374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.726354 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.778406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.778381 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:34.778500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:34.778456 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfbx\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:35.062485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.062456 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:35.356737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.356643 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899d8915-3cf4-4bb3-bb2f-4aee0920417e" path="/var/lib/kubelet/pods/899d8915-3cf4-4bb3-bb2f-4aee0920417e/volumes" Nov 04 01:21:35.487737 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:35.487706 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6be3863_9fd0_4c96_837a_0d599552105e.slice/crio-cb78b4be1a7edf6feb424616c348e36cd80391f8711cf92fd202898b44ded058 WatchSource:0}: Error finding container cb78b4be1a7edf6feb424616c348e36cd80391f8711cf92fd202898b44ded058: Status 404 returned error can't find the container with id cb78b4be1a7edf6feb424616c348e36cd80391f8711cf92fd202898b44ded058 Nov 04 01:21:35.488754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.488731 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:21:35.490278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.490241 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:35.490381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.490303 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:35.490381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.490333 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:35.721809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.721724 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" event={"ID":"f6be3863-9fd0-4c96-837a-0d599552105e","Type":"ContainerStarted","Data":"e44f9604447ca9ec1005de36113ab3ade7ff2c92b277be2e66f2c2ad07a396fb"} Nov 04 01:21:35.721809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.721790 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" event={"ID":"f6be3863-9fd0-4c96-837a-0d599552105e","Type":"ContainerStarted","Data":"cb78b4be1a7edf6feb424616c348e36cd80391f8711cf92fd202898b44ded058"} Nov 04 01:21:35.857133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.855343 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podStartSLOduration=1.855323264 podStartE2EDuration="1.855323264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:35.83663035 +0000 UTC m=+1464.595292512" watchObservedRunningTime="2026-11-04 01:21:35.855323264 +0000 UTC m=+1464.613985427" Nov 04 01:21:35.922032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:35.922005 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:36.059280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.059117 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\") pod \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " Nov 04 01:21:36.059459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.059344 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\") pod \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " Nov 04 01:21:36.059459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.059412 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\" (UniqueName: \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\") pod \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\" (UID: \"d3151800-d67b-4e59-93fd-ec9eb5e23f74\") " Nov 04 01:21:36.065004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.061972 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca" pod "d3151800-d67b-4e59-93fd-ec9eb5e23f74" (UID: "d3151800-d67b-4e59-93fd-ec9eb5e23f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:36.065004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.062753 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:36.065004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.064198 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:36.065004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.064245 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:36.079619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.079583 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv" pod "d3151800-d67b-4e59-93fd-ec9eb5e23f74" (UID: "d3151800-d67b-4e59-93fd-ec9eb5e23f74"). InnerVolumeSpecName "kube-api-access-4s5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:36.079780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.079730 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca" pod "d3151800-d67b-4e59-93fd-ec9eb5e23f74" (UID: "d3151800-d67b-4e59-93fd-ec9eb5e23f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:36.160829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.160790 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5gv\" (UniqueName: \"kubernetes.io/projected/d3151800-d67b-4e59-93fd-ec9eb5e23f74-kube-api-access-4s5gv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:36.160829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.160822 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:36.160829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.160834 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3151800-d67b-4e59-93fd-ec9eb5e23f74-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:36.760530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.760500 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" Nov 04 01:21:36.761018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.760507 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c" event={"ID":"d3151800-d67b-4e59-93fd-ec9eb5e23f74","Type":"ContainerDied","Data":"c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066"} Nov 04 01:21:36.761018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:36.760616 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c188841657807ec7687ce82e4d4048383c870d3c130dab20cc314abd51373066" Nov 04 01:21:37.063430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:37.063406 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:37.063576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:37.063454 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:38.063073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:38.063028 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:38.063441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:38.063097 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:39.057892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.057854 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg"] Nov 04 01:21:39.058931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.058904 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d3151800-d67b-4e59-93fd-ec9eb5e23f74" containerName="revoke-keys" Nov 04 01:21:39.059056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.058933 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3151800-d67b-4e59-93fd-ec9eb5e23f74" containerName="revoke-keys" Nov 04 01:21:39.059158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.059143 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3151800-d67b-4e59-93fd-ec9eb5e23f74" containerName="revoke-keys" Nov 04 01:21:39.063440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.063406 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:39.063853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.063461 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:39.063928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.063859 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.096165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.096132 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.096329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.096307 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm58\" (UniqueName: \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.096388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.096376 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.111702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.111677 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg"] Nov 04 01:21:39.163017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.153374 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:21:39.163017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.153480 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:39.198314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.198281 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm58\" (UniqueName: \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.198447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.198379 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.198509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.198494 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.203025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.199366 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.203025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.202735 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.259173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.259104 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm58\" (UniqueName: \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\") pod \"maas-api-revoke-sub-dce3bf3c-ljqlg\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.379120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.379086 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:39.787790 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:39.787755 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4f115_36b5_4fea_838a_958f88ef72f8.slice/crio-fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085 WatchSource:0}: Error finding container fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085: Status 404 returned error can't find the container with id fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085 Nov 04 01:21:39.789633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:39.789605 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg"] Nov 04 01:21:40.063278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.063251 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:40.063421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.063306 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:40.797477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.797447 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255"] Nov 04 01:21:40.802488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.802463 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" event={"ID":"bcb4f115-36b5-4fea-838a-958f88ef72f8","Type":"ContainerStarted","Data":"0020c36edc67ae9ead94f56560bef9a21c1f7fc8016315564a858ecfabf6b109"} Nov 04 01:21:40.802599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.802496 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" event={"ID":"bcb4f115-36b5-4fea-838a-958f88ef72f8","Type":"ContainerStarted","Data":"fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085"} Nov 04 01:21:40.802599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.802563 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.821634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.821608 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.821759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.821683 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.822062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.822037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqbz\" (UniqueName: \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.861255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.861224 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255"] Nov 04 01:21:40.923599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.923569 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.923697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.923668 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.923791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.923776 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqbz\" (UniqueName: \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.924554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.924527 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.925826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.925803 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:40.966582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.966518 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" podStartSLOduration=1.966500948 podStartE2EDuration="1.966500948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:40.962177937 +0000 UTC m=+1469.720840101" watchObservedRunningTime="2026-11-04 01:21:40.966500948 +0000 UTC m=+1469.725163110" Nov 04 01:21:40.980625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:40.980596 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqbz\" (UniqueName: \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\") pod \"maas-api-revoke-sub-88a55a0f-jb255\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:41.063725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.063694 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:41.063895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.063747 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:41.117638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.117601 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:41.143003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.128286 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs"] Nov 04 01:21:41.170215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.169366 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.195456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.195426 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs"] Nov 04 01:21:41.278272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.269320 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ngf\" (UniqueName: \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.278272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.269464 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.278272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.269504 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.371885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.371795 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ngf\" (UniqueName: \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.372068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.371973 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.372068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.372047 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.372847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.372819 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.386067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.376283 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.453002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.449082 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ngf\" (UniqueName: \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\") pod \"maas-api-revoke-sub-fe00dfd5-dtmhs\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.497579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.497540 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:41.588411 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:41.588318 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d324d7_9570_4737_9c6d_f332d2f77a1e.slice/crio-8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200 WatchSource:0}: Error finding container 8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200: Status 404 returned error can't find the container with id 8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200 Nov 04 01:21:41.592211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.591742 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255"] Nov 04 01:21:41.811053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.811012 4486 generic.go:356] "Generic (PLEG): container finished" podID="bcb4f115-36b5-4fea-838a-958f88ef72f8" containerID="0020c36edc67ae9ead94f56560bef9a21c1f7fc8016315564a858ecfabf6b109" exitCode=0 Nov 04 01:21:41.811441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.811101 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" event={"ID":"bcb4f115-36b5-4fea-838a-958f88ef72f8","Type":"ContainerDied","Data":"0020c36edc67ae9ead94f56560bef9a21c1f7fc8016315564a858ecfabf6b109"} Nov 04 01:21:41.812732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.812709 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" event={"ID":"a0d324d7-9570-4737-9c6d-f332d2f77a1e","Type":"ContainerStarted","Data":"8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200"} Nov 04 01:21:41.887431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:41.887404 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs"] Nov 04 01:21:41.887939 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:41.887914 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1ee5ac_7627_43b7_b72e_131ab52070e5.slice/crio-e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083 WatchSource:0}: Error finding container e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083: Status 404 returned error can't find the container with id e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083 Nov 04 01:21:42.064367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.064328 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:21:42.064501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.064377 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:21:42.827405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.827365 4486 generic.go:356] "Generic (PLEG): container finished" podID="cf1ee5ac-7627-43b7-b72e-131ab52070e5" containerID="549fa0a8f2174821e3b6c044b21c824e6c36e4b542e9099c61333ab1b9937376" exitCode=0 Nov 04 01:21:42.827829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.827425 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" event={"ID":"cf1ee5ac-7627-43b7-b72e-131ab52070e5","Type":"ContainerDied","Data":"549fa0a8f2174821e3b6c044b21c824e6c36e4b542e9099c61333ab1b9937376"} Nov 04 01:21:42.827829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.827474 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" event={"ID":"cf1ee5ac-7627-43b7-b72e-131ab52070e5","Type":"ContainerStarted","Data":"e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083"} Nov 04 01:21:42.829740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.829713 4486 generic.go:356] "Generic (PLEG): container finished" podID="a0d324d7-9570-4737-9c6d-f332d2f77a1e" containerID="fe6580a4b67fd492dd45b5696b33b5e49ac8cca8a828fa02c4bbd3ffe865465d" exitCode=0 Nov 04 01:21:42.829847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:42.829788 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" event={"ID":"a0d324d7-9570-4737-9c6d-f332d2f77a1e","Type":"ContainerDied","Data":"fe6580a4b67fd492dd45b5696b33b5e49ac8cca8a828fa02c4bbd3ffe865465d"} Nov 04 01:21:43.000438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.000371 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:43.068392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.068358 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:43.068677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.068654 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:43.069512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.069493 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:21:43.100784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.100753 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\") pod \"bcb4f115-36b5-4fea-838a-958f88ef72f8\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " Nov 04 01:21:43.100962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.100808 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\" (UniqueName: \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\") pod \"bcb4f115-36b5-4fea-838a-958f88ef72f8\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " Nov 04 01:21:43.100962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.100847 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\") pod \"bcb4f115-36b5-4fea-838a-958f88ef72f8\" (UID: \"bcb4f115-36b5-4fea-838a-958f88ef72f8\") " Nov 04 01:21:43.101339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.101312 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca" pod "bcb4f115-36b5-4fea-838a-958f88ef72f8" (UID: "bcb4f115-36b5-4fea-838a-958f88ef72f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:43.101584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.101565 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:43.103036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.103011 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca" pod "bcb4f115-36b5-4fea-838a-958f88ef72f8" (UID: "bcb4f115-36b5-4fea-838a-958f88ef72f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:43.103354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.103333 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58" pod "bcb4f115-36b5-4fea-838a-958f88ef72f8" (UID: "bcb4f115-36b5-4fea-838a-958f88ef72f8"). InnerVolumeSpecName "kube-api-access-fnm58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:43.204191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.204160 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb4f115-36b5-4fea-838a-958f88ef72f8-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:43.204331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.204205 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm58\" (UniqueName: \"kubernetes.io/projected/bcb4f115-36b5-4fea-838a-958f88ef72f8-kube-api-access-fnm58\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:43.213283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.213254 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc"] Nov 04 01:21:43.214033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.214006 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bcb4f115-36b5-4fea-838a-958f88ef72f8" containerName="revoke-keys" Nov 04 01:21:43.214033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.214029 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb4f115-36b5-4fea-838a-958f88ef72f8" containerName="revoke-keys" Nov 04 01:21:43.214213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.214202 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb4f115-36b5-4fea-838a-958f88ef72f8" containerName="revoke-keys" Nov 04 01:21:43.217738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.217723 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.270319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.270247 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc"] Nov 04 01:21:43.308019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.305054 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.308019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.305296 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.308019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.305383 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdv6p\" (UniqueName: \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.406917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.406883 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.407085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.406966 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdv6p\" (UniqueName: \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.407085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.407029 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.407550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.407526 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.409420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.409396 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.461917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.461893 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdv6p\" (UniqueName: \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\") pod \"maas-api-revoke-sub-f110a619-2rlhc\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.534403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.534342 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:43.833086 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:21:43.832955 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:21:43.841437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.841412 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" Nov 04 01:21:43.841604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.841451 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg" event={"ID":"bcb4f115-36b5-4fea-838a-958f88ef72f8","Type":"ContainerDied","Data":"fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085"} Nov 04 01:21:43.841604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.841501 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe59730a0fe55204dad01596cc4aa474fd15e28d9d01920c1288b4abafe17085" Nov 04 01:21:43.921893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.921806 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc"] Nov 04 01:21:43.991604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:43.991135 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:44.042641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.042616 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:44.121052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.120915 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\") pod \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " Nov 04 01:21:44.121052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.120975 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\") pod \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " Nov 04 01:21:44.121292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.121132 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\" (UniqueName: \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\") pod \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\" (UID: \"a0d324d7-9570-4737-9c6d-f332d2f77a1e\") " Nov 04 01:21:44.122529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.122500 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca" pod "a0d324d7-9570-4737-9c6d-f332d2f77a1e" (UID: "a0d324d7-9570-4737-9c6d-f332d2f77a1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:44.124541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.124514 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz" pod "a0d324d7-9570-4737-9c6d-f332d2f77a1e" (UID: "a0d324d7-9570-4737-9c6d-f332d2f77a1e"). InnerVolumeSpecName "kube-api-access-hpqbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:44.124740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.124707 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca" pod "a0d324d7-9570-4737-9c6d-f332d2f77a1e" (UID: "a0d324d7-9570-4737-9c6d-f332d2f77a1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:44.223125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.223091 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\" (UniqueName: \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\") pod \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " Nov 04 01:21:44.223321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.223303 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\") pod \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " Nov 04 01:21:44.223435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.223420 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\") pod \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\" (UID: \"cf1ee5ac-7627-43b7-b72e-131ab52070e5\") " Nov 04 01:21:44.223956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.223923 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca" pod "cf1ee5ac-7627-43b7-b72e-131ab52070e5" (UID: "cf1ee5ac-7627-43b7-b72e-131ab52070e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:44.224130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.224108 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.224200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.224135 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.224200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.224149 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d324d7-9570-4737-9c6d-f332d2f77a1e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.224200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.224162 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqbz\" (UniqueName: \"kubernetes.io/projected/a0d324d7-9570-4737-9c6d-f332d2f77a1e-kube-api-access-hpqbz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.227049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.226967 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf" pod "cf1ee5ac-7627-43b7-b72e-131ab52070e5" (UID: "cf1ee5ac-7627-43b7-b72e-131ab52070e5"). InnerVolumeSpecName "kube-api-access-b9ngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:44.227626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.227581 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca" pod "cf1ee5ac-7627-43b7-b72e-131ab52070e5" (UID: "cf1ee5ac-7627-43b7-b72e-131ab52070e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:44.325622 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.325580 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ngf\" (UniqueName: \"kubernetes.io/projected/cf1ee5ac-7627-43b7-b72e-131ab52070e5-kube-api-access-b9ngf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.325622 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.325611 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf1ee5ac-7627-43b7-b72e-131ab52070e5-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:44.623222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.623183 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn"] Nov 04 01:21:44.623899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.623878 4486 cpu_manager.go:414] "containerMap: removing container" podUID="cf1ee5ac-7627-43b7-b72e-131ab52070e5" containerName="revoke-keys" Nov 04 01:21:44.624044 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.623901 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1ee5ac-7627-43b7-b72e-131ab52070e5" containerName="revoke-keys" Nov 04 01:21:44.624044 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.623927 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a0d324d7-9570-4737-9c6d-f332d2f77a1e" containerName="revoke-keys" Nov 04 01:21:44.624044 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.623935 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d324d7-9570-4737-9c6d-f332d2f77a1e" containerName="revoke-keys" Nov 04 01:21:44.624197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.624138 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1ee5ac-7627-43b7-b72e-131ab52070e5" containerName="revoke-keys" Nov 04 01:21:44.624197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.624157 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d324d7-9570-4737-9c6d-f332d2f77a1e" containerName="revoke-keys" Nov 04 01:21:44.627960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.627938 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.681346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.681313 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn"] Nov 04 01:21:44.731939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.731903 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.732105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.732060 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.732183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.732131 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mskt\" (UniqueName: \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.833100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.833062 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.833521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.833151 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.833521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.833233 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mskt\" (UniqueName: \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.833869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.833844 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.835471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.835451 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.852746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.852723 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" Nov 04 01:21:44.852746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.852728 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs" event={"ID":"cf1ee5ac-7627-43b7-b72e-131ab52070e5","Type":"ContainerDied","Data":"e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083"} Nov 04 01:21:44.852938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.852786 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e788807e741d58f3a65adba50f0382d94a6697f78e8df2c569ad7e5a33816083" Nov 04 01:21:44.855581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.855559 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" Nov 04 01:21:44.855719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.855583 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255" event={"ID":"a0d324d7-9570-4737-9c6d-f332d2f77a1e","Type":"ContainerDied","Data":"8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200"} Nov 04 01:21:44.855719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.855644 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8842da10ca0ebac31fb306e4773a91ae7eb585afaac53c1a79425bb14b24e200" Nov 04 01:21:44.857664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.857638 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" event={"ID":"7fb9f6e1-2995-4a2f-9923-aea53664f57f","Type":"ContainerStarted","Data":"155babbce3f04f1ba6a604c4fd9e633cdf1d90960fa686199e185fc1aae36af7"} Nov 04 01:21:44.857771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.857686 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" event={"ID":"7fb9f6e1-2995-4a2f-9923-aea53664f57f","Type":"ContainerStarted","Data":"c29bd2fb751f40260189070b1916b756379fcc4fa985d14fdea9ff73dcb5eec2"} Nov 04 01:21:44.888753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.888696 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mskt\" (UniqueName: \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\") pod \"maas-api-revoke-sub-c5aab579-9cfsn\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.945318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.945296 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:44.972270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:44.972226 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" podStartSLOduration=1.972210738 podStartE2EDuration="1.972210738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:44.969156603 +0000 UTC m=+1473.727818744" watchObservedRunningTime="2026-11-04 01:21:44.972210738 +0000 UTC m=+1473.730872913" Nov 04 01:21:45.334808 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:45.334774 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dbe76d_3225_46bf_8f1b_fef4b74dc31c.slice/crio-6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f WatchSource:0}: Error finding container 6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f: Status 404 returned error can't find the container with id 6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f Nov 04 01:21:45.335991 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.335958 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn"] Nov 04 01:21:45.869532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.869494 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" event={"ID":"67dbe76d-3225-46bf-8f1b-fef4b74dc31c","Type":"ContainerStarted","Data":"cc5a2740c3b764b5fa989e1cb090e9624678c53e84eefd23ff3f79caf8ede985"} Nov 04 01:21:45.869949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.869550 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" event={"ID":"67dbe76d-3225-46bf-8f1b-fef4b74dc31c","Type":"ContainerStarted","Data":"6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f"} Nov 04 01:21:45.871355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.871330 4486 generic.go:356] "Generic (PLEG): container finished" podID="7fb9f6e1-2995-4a2f-9923-aea53664f57f" containerID="155babbce3f04f1ba6a604c4fd9e633cdf1d90960fa686199e185fc1aae36af7" exitCode=0 Nov 04 01:21:45.871473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.871409 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" event={"ID":"7fb9f6e1-2995-4a2f-9923-aea53664f57f","Type":"ContainerDied","Data":"155babbce3f04f1ba6a604c4fd9e633cdf1d90960fa686199e185fc1aae36af7"} Nov 04 01:21:45.983376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:45.983286 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" podStartSLOduration=1.983271115 podStartE2EDuration="1.983271115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:45.978976465 +0000 UTC m=+1474.737638628" watchObservedRunningTime="2026-11-04 01:21:45.983271115 +0000 UTC m=+1474.741933276" Nov 04 01:21:46.881412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:46.881374 4486 generic.go:356] "Generic (PLEG): container finished" podID="67dbe76d-3225-46bf-8f1b-fef4b74dc31c" containerID="cc5a2740c3b764b5fa989e1cb090e9624678c53e84eefd23ff3f79caf8ede985" exitCode=0 Nov 04 01:21:46.881813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:46.881460 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" event={"ID":"67dbe76d-3225-46bf-8f1b-fef4b74dc31c","Type":"ContainerDied","Data":"cc5a2740c3b764b5fa989e1cb090e9624678c53e84eefd23ff3f79caf8ede985"} Nov 04 01:21:47.072861 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.072833 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:47.265158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.265079 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\") pod \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " Nov 04 01:21:47.265304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.265264 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\") pod \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " Nov 04 01:21:47.265444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.265426 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\" (UniqueName: \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\") pod \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\" (UID: \"7fb9f6e1-2995-4a2f-9923-aea53664f57f\") " Nov 04 01:21:47.265504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.265485 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca" pod "7fb9f6e1-2995-4a2f-9923-aea53664f57f" (UID: "7fb9f6e1-2995-4a2f-9923-aea53664f57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:47.266068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.266038 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:47.267550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.267513 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p" pod "7fb9f6e1-2995-4a2f-9923-aea53664f57f" (UID: "7fb9f6e1-2995-4a2f-9923-aea53664f57f"). InnerVolumeSpecName "kube-api-access-jdv6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:47.267673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.267560 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca" pod "7fb9f6e1-2995-4a2f-9923-aea53664f57f" (UID: "7fb9f6e1-2995-4a2f-9923-aea53664f57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:47.368087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.368044 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9f6e1-2995-4a2f-9923-aea53664f57f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:47.368087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.368087 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdv6p\" (UniqueName: \"kubernetes.io/projected/7fb9f6e1-2995-4a2f-9923-aea53664f57f-kube-api-access-jdv6p\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:47.891569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.891531 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" event={"ID":"7fb9f6e1-2995-4a2f-9923-aea53664f57f","Type":"ContainerDied","Data":"c29bd2fb751f40260189070b1916b756379fcc4fa985d14fdea9ff73dcb5eec2"} Nov 04 01:21:47.891966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.891590 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29bd2fb751f40260189070b1916b756379fcc4fa985d14fdea9ff73dcb5eec2" Nov 04 01:21:47.891966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:47.891545 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc" Nov 04 01:21:48.046695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.046667 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:48.179763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.179679 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\") pod \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " Nov 04 01:21:48.179763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.179750 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\" (UniqueName: \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\") pod \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " Nov 04 01:21:48.180035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.180016 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\") pod \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\" (UID: \"67dbe76d-3225-46bf-8f1b-fef4b74dc31c\") " Nov 04 01:21:48.181111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.181080 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca" pod "67dbe76d-3225-46bf-8f1b-fef4b74dc31c" (UID: "67dbe76d-3225-46bf-8f1b-fef4b74dc31c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:48.182865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.182816 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca" pod "67dbe76d-3225-46bf-8f1b-fef4b74dc31c" (UID: "67dbe76d-3225-46bf-8f1b-fef4b74dc31c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:48.183440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.183412 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt" pod "67dbe76d-3225-46bf-8f1b-fef4b74dc31c" (UID: "67dbe76d-3225-46bf-8f1b-fef4b74dc31c"). InnerVolumeSpecName "kube-api-access-8mskt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:48.282160 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.282132 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:48.282160 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.282159 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:48.282313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.282171 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mskt\" (UniqueName: \"kubernetes.io/projected/67dbe76d-3225-46bf-8f1b-fef4b74dc31c-kube-api-access-8mskt\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:48.341837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.341794 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d"] Nov 04 01:21:48.342802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.342782 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7fb9f6e1-2995-4a2f-9923-aea53664f57f" containerName="revoke-keys" Nov 04 01:21:48.342890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.342806 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb9f6e1-2995-4a2f-9923-aea53664f57f" containerName="revoke-keys" Nov 04 01:21:48.342890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.342831 4486 cpu_manager.go:414] "containerMap: removing container" podUID="67dbe76d-3225-46bf-8f1b-fef4b74dc31c" containerName="revoke-keys" Nov 04 01:21:48.342890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.342840 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dbe76d-3225-46bf-8f1b-fef4b74dc31c" containerName="revoke-keys" Nov 04 01:21:48.343065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.343024 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb9f6e1-2995-4a2f-9923-aea53664f57f" containerName="revoke-keys" Nov 04 01:21:48.343065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.343041 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dbe76d-3225-46bf-8f1b-fef4b74dc31c" containerName="revoke-keys" Nov 04 01:21:48.347762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.347743 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.383752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.383713 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.383874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.383815 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn54l\" (UniqueName: \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.383874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.383854 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.395892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.395863 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d"] Nov 04 01:21:48.485212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.485125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.485350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.485259 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn54l\" (UniqueName: \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.485350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.485310 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.486121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.486091 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.487726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.487705 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.542219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.542183 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn54l\" (UniqueName: \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\") pod \"maas-api-revoke-sub-e454888a-2np9d\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.583932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.583894 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:21:48.588861 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.588836 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.681009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.673519 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:48.700952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.700915 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:21:48.793754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.793721 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.793904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.793840 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.793904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.793886 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794062 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794105 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794144 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794163 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.794257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.794198 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kmc\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.846440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.846410 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l"] Nov 04 01:21:48.853389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.853362 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.895410 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895378 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895421 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895467 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kmc\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895529 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895589 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895622 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895679 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895776 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.895812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895781 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7kv\" (UniqueName: \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895862 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895890 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895925 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.895970 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.896084 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.896469 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.896745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.896638 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.897612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.897585 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.900304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.900259 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.907109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.906970 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.907109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.907047 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l"] Nov 04 01:21:48.908783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.908759 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" event={"ID":"67dbe76d-3225-46bf-8f1b-fef4b74dc31c","Type":"ContainerDied","Data":"6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f"} Nov 04 01:21:48.908875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.908806 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff549d0ffceab4946191fe63cf66e6c0717f35eea05e203bea92ca46b39956f" Nov 04 01:21:48.908935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.908887 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn" Nov 04 01:21:48.912782 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.912760 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:21:48.918479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.918457 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.925952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.925914 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:21:48.938953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.937239 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.966739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.966707 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kmc\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.971446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.971406 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\") pod \"e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:48.974825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.974798 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:21:48.997755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997727 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.997877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997773 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.997877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997804 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxfr\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.997877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997830 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.997877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997857 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997938 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.997965 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998022 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998055 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998091 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998116 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998334 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7kv\" (UniqueName: \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998381 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998411 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998591 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998638 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998671 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.998741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998717 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.999154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998757 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.999154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998921 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfr7\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:48.999154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.998953 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:48.999864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:48.999825 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:49.003668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.003616 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:49.051268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.051218 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:21:49.055085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.055054 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7kv\" (UniqueName: \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\") pod \"maas-api-revoke-sub-5ac6d1e3-hmw2l\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:49.087951 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:49.086550 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6658141e_666b_4e5f_b6a3_ebb73346e0f3.slice/crio-6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e WatchSource:0}: Error finding container 6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e: Status 404 returned error can't find the container with id 6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e Nov 04 01:21:49.089175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.089153 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d"] Nov 04 01:21:49.099946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.099922 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfr7\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.099960 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100028 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100065 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100096 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxfr\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100122 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100150 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100197 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100242 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100273 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100311 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100336 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100379 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100397 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100725 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.100827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100802 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.100964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100939 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.101043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.100979 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.101043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101019 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.101159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101071 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.101159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101117 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.101267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101156 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.101267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101249 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.101376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101022 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.101684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.101631 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.102620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.102595 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.102760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.102733 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.103217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.103188 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.103309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.103292 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.104231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.104207 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.104335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.104231 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.105473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.105448 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.155272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.155241 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.156280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.156256 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.156434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.156411 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfr7\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.156703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.156682 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxfr\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\") pod \"e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.171743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.171603 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:49.204368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.204345 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:49.244111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.242116 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:49.281534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.278232 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:49.564298 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:49.564268 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b565503_8053_4944_9cfc_a9a068ba8cb1.slice/crio-4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03 WatchSource:0}: Error finding container 4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03: Status 404 returned error can't find the container with id 4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03 Nov 04 01:21:49.566610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.566587 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l"] Nov 04 01:21:49.617494 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:49.617463 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc52389_ac0b_4510_8364_9868250cc9e6.slice/crio-0fc3c1666d324fec02c0c09150663587a344fa6d96673c9410b876bef160ad73 WatchSource:0}: Error finding container 0fc3c1666d324fec02c0c09150663587a344fa6d96673c9410b876bef160ad73: Status 404 returned error can't find the container with id 0fc3c1666d324fec02c0c09150663587a344fa6d96673c9410b876bef160ad73 Nov 04 01:21:49.620039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.619977 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.620135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.620094 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.620135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.620127 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.620809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.620781 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:21:49.684338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.684308 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:21:49.714425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.714400 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:21:49.736337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.735667 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.736337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.735761 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.736337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.735820 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.739711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.739624 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.739711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.739693 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.739821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.739737 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:49.897962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.897922 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:21:49.904647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.904565 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.978336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.950677 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.978336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.950755 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.978336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.950813 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.978336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.950855 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9xg\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.978336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.950889 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.994755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.989312 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.994755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.989431 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.994755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.989493 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:49.994755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:49.989557 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.022059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.022030 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:21:50.042317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.042288 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.056369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.056325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" event={"ID":"07be9abf-2036-4221-805a-b1b856ac1be3","Type":"ContainerStarted","Data":"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124"} Nov 04 01:21:50.056586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.056569 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" event={"ID":"07be9abf-2036-4221-805a-b1b856ac1be3","Type":"ContainerStarted","Data":"f4686cc3b08128f34734bb525e3004bb4103e29f6d12d3abe00e61d100db29bb"} Nov 04 01:21:50.059546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.059519 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" event={"ID":"6658141e-666b-4e5f-b6a3-ebb73346e0f3","Type":"ContainerStarted","Data":"10d2e7bca2ed97a700e52a1ada66d05d3196ad8cfe970ecc1e0480abd3e748a2"} Nov 04 01:21:50.059704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.059690 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" event={"ID":"6658141e-666b-4e5f-b6a3-ebb73346e0f3","Type":"ContainerStarted","Data":"6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e"} Nov 04 01:21:50.062426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.062405 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" event={"ID":"545a415b-07e6-425b-a489-f269781812f2","Type":"ContainerStarted","Data":"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca"} Nov 04 01:21:50.062563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.062548 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" event={"ID":"545a415b-07e6-425b-a489-f269781812f2","Type":"ContainerStarted","Data":"cd6e5d80187946844ef962a158cdf34873b7b5dce731915ff0e9a1394e340265"} Nov 04 01:21:50.085666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.071455 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" event={"ID":"dfc52389-ac0b-4510-8364-9868250cc9e6","Type":"ContainerStarted","Data":"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1"} Nov 04 01:21:50.085666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.071512 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" event={"ID":"dfc52389-ac0b-4510-8364-9868250cc9e6","Type":"ContainerStarted","Data":"0fc3c1666d324fec02c0c09150663587a344fa6d96673c9410b876bef160ad73"} Nov 04 01:21:50.085666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.074116 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" event={"ID":"9b565503-8053-4944-9cfc-a9a068ba8cb1","Type":"ContainerStarted","Data":"d8eba2e1aa08a43f2a1804387f48ec320041e8bb6e304d6d698a49ba487eeff9"} Nov 04 01:21:50.085666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.074157 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" event={"ID":"9b565503-8053-4944-9cfc-a9a068ba8cb1","Type":"ContainerStarted","Data":"4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03"} Nov 04 01:21:50.085666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.076745 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:21:50.091670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.091642 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:21:50.092428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092406 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092468 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092517 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092557 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9xg\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092589 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092666 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092716 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092755 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.092818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.092805 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.093331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.093296 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.093539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.093520 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.096126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.096103 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.097107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.097083 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.097889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.097867 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.098269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.098250 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.099620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.099598 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.157000 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.151966 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.165554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.165437 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9xg\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.194022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.193955 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194112 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194133 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194233 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194357 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194403 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194419 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194477 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.194610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.194578 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.204868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.204845 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:21:50.206258 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.206231 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.206375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.206294 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:50.244663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.244633 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:21:50.244838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.244744 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.244838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.244793 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:50.280492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.280431 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podStartSLOduration=2.280414368 podStartE2EDuration="2.280414368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.279073326 +0000 UTC m=+1479.037735488" watchObservedRunningTime="2026-11-04 01:21:50.280414368 +0000 UTC m=+1479.039076531" Nov 04 01:21:50.281689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.281665 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:21:50.283247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.283217 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.283345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.283279 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.295823 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.295875 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.295901 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.295947 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.296133 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.296208 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.296309 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.296338 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.296594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.296416 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.297187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.297074 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.297765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.297742 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.298034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.297969 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.298467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.298449 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.299012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.298975 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.301555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.301533 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.303506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.303478 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.342436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.342363 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:50.380850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.380783 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.384169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.384140 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\") pod \"e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.396831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.396784 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" podStartSLOduration=2.3967683490000002 podStartE2EDuration="2.396768349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.391411639 +0000 UTC m=+1479.150073801" watchObservedRunningTime="2026-11-04 01:21:50.396768349 +0000 UTC m=+1479.155430513" Nov 04 01:21:50.435215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.434918 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:50.518170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.518106 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podStartSLOduration=2.5180881619999997 podStartE2EDuration="2.518088162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.507686546 +0000 UTC m=+1479.266348708" watchObservedRunningTime="2026-11-04 01:21:50.518088162 +0000 UTC m=+1479.276750329" Nov 04 01:21:50.628669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.628563 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" podStartSLOduration=2.628547425 podStartE2EDuration="2.628547425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.623958438 +0000 UTC m=+1479.382620601" watchObservedRunningTime="2026-11-04 01:21:50.628547425 +0000 UTC m=+1479.387209591" Nov 04 01:21:50.658401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.658366 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:21:50.674515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.674489 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.750931 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751016 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751054 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751105 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751108 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751179 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751356 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751442 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751487 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.751534 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8sw\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.764060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.761471 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podStartSLOduration=2.761454745 podStartE2EDuration="2.761454745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:50.759548933 +0000 UTC m=+1479.518211096" watchObservedRunningTime="2026-11-04 01:21:50.761454745 +0000 UTC m=+1479.520116909" Nov 04 01:21:50.784648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.782872 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.784648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.783034 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.784648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.783095 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.797315 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.797276 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856127 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856187 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856223 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856268 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856326 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856390 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856467 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856500 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.856539 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8sw\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.857750 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.858086 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.867030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.858333 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.873227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.871795 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.873227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.872187 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.873227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.873155 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.874891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.874861 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.878154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.878129 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:21:50.886522 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:50.886499 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d05d797_e34e_4cb3_959c_c6e3f75235b9.slice/crio-7c0a10093eb0e8fa21cae76c5ef8b933aab5db25517e2bd5169ed3c545e056f9 WatchSource:0}: Error finding container 7c0a10093eb0e8fa21cae76c5ef8b933aab5db25517e2bd5169ed3c545e056f9: Status 404 returned error can't find the container with id 7c0a10093eb0e8fa21cae76c5ef8b933aab5db25517e2bd5169ed3c545e056f9 Nov 04 01:21:50.889112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.889081 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.889212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.889152 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.889212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.889184 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:50.911127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.911108 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8sw\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.926387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.926361 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\") pod \"e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:50.990470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:50.990238 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:51.181067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.180944 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" event={"ID":"7d05d797-e34e-4cb3-959c-c6e3f75235b9","Type":"ContainerStarted","Data":"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c"} Nov 04 01:21:51.181323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.181299 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" event={"ID":"7d05d797-e34e-4cb3-959c-c6e3f75235b9","Type":"ContainerStarted","Data":"7c0a10093eb0e8fa21cae76c5ef8b933aab5db25517e2bd5169ed3c545e056f9"} Nov 04 01:21:51.217008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.205422 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.217008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.205483 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:51.489092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.371653 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.489092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.371711 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:51.489092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.455341 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.489092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.455395 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:51.489092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.469091 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podStartSLOduration=2.469072016 podStartE2EDuration="2.469072016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:51.413629832 +0000 UTC m=+1480.172291996" watchObservedRunningTime="2026-11-04 01:21:51.469072016 +0000 UTC m=+1480.227734179" Nov 04 01:21:51.961122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:51.961086 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:21:52.121554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.114310 4486 generic.go:356] "Generic (PLEG): container finished" podID="6658141e-666b-4e5f-b6a3-ebb73346e0f3" containerID="10d2e7bca2ed97a700e52a1ada66d05d3196ad8cfe970ecc1e0480abd3e748a2" exitCode=0 Nov 04 01:21:52.121554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.114818 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" event={"ID":"6658141e-666b-4e5f-b6a3-ebb73346e0f3","Type":"ContainerDied","Data":"10d2e7bca2ed97a700e52a1ada66d05d3196ad8cfe970ecc1e0480abd3e748a2"} Nov 04 01:21:52.137821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.137786 4486 generic.go:356] "Generic (PLEG): container finished" podID="9b565503-8053-4944-9cfc-a9a068ba8cb1" containerID="d8eba2e1aa08a43f2a1804387f48ec320041e8bb6e304d6d698a49ba487eeff9" exitCode=0 Nov 04 01:21:52.138354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.138324 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" event={"ID":"9b565503-8053-4944-9cfc-a9a068ba8cb1","Type":"ContainerDied","Data":"d8eba2e1aa08a43f2a1804387f48ec320041e8bb6e304d6d698a49ba487eeff9"} Nov 04 01:21:52.141009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.140905 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:21:52.163846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.149158 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" event={"ID":"d09a2489-14fb-408f-9842-1736efb0e3d4","Type":"ContainerStarted","Data":"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995"} Nov 04 01:21:52.163846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.149233 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" event={"ID":"d09a2489-14fb-408f-9842-1736efb0e3d4","Type":"ContainerStarted","Data":"26fec395c3ceba29079afd6bda1d8c26c75d61c5af96687e7acf2ba40ace0520"} Nov 04 01:21:52.278589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.278494 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.337308 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.337245 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:52.338128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.296109 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.338332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.338293 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:52.340142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.338630 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.463235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.463170 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.759638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.641248 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:52.759638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.737209 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:21:52.759638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.738213 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.759638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.759281 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:52.787011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.761811 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.787011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.761882 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:52.909913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.909849 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podStartSLOduration=3.909828711 podStartE2EDuration="3.909828711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:52.898835021 +0000 UTC m=+1481.657497183" watchObservedRunningTime="2026-11-04 01:21:52.909828711 +0000 UTC m=+1481.668490873" Nov 04 01:21:52.932698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.932627 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:52.954417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.954376 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:52.954679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:52.954657 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:21:53.224150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.224112 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.224726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.224695 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:53.251460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.251422 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.272892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.272852 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:53.275373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.252468 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.275829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.252428 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:21:53.295737 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.295700 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" event={"ID":"440f9d59-95a2-492f-af05-cc83fd171f83","Type":"ContainerStarted","Data":"f47f87511a6c3c0456da46dc6a739c7ec0fb512ee124d68f1802cc1233742361"} Nov 04 01:21:53.447496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.358213 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.447496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.358272 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:53.448341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.447921 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.449885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.448847 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:21:53.449885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.449781 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.449885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.449823 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:53.450438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.450418 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:53.450593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.450571 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:53.713348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.713249 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:21:53.739011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.714361 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.807202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.803277 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.807202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:53.803798 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:21:54.044074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.034912 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:21:54.227037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.218346 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.227037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.218408 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:54.277089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.263380 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.277089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.263437 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:54.297062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.295565 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.297062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.295871 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.300135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.300109 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:21:54.335777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.335753 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:21:54.336096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.309928 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.343084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.343049 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:54.343220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.334145 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.453003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.447811 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.453003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.447871 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:54.453003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.448939 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.453003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.449284 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:21:54.673229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.673170 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.695039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.693152 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:54.695039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.693199 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:54.793960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.793918 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:54.796813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.796788 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:21:54.826802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.826760 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.833131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.833092 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.028173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:54.835205 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:21:55.057611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.057567 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" event={"ID":"440f9d59-95a2-492f-af05-cc83fd171f83","Type":"ContainerStarted","Data":"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9"} Nov 04 01:21:55.067009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.058410 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.090205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.090173 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:21:55.258281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.258246 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.427379 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.296432 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.287321 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.377623 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.289393 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.434450 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.401494 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.434502 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.296491 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:21:55.445054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.401824 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:21:55.483139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.466693 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:21:55.540021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.513414 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:55.540021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:55.513479 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:56.019191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.018777 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:21:56.019191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.018834 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:21:56.019666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.019526 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.077200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.076645 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:56.085664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.085631 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.129037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.112281 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:56.157008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.145103 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:21:56.261140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.150161 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.261140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.223344 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:21:56.261140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.224065 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:21:56.261140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.224738 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.338077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.287074 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.338077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.287142 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:56.401169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.401136 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:56.458705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.417600 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.459001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.458948 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:56.507249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.507174 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.507574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.507526 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:56.507855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.507811 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.508026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.508006 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:56.652074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.620160 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\" (UniqueName: \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\") pod \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " Nov 04 01:21:56.661325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.661290 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") pod \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " Nov 04 01:21:56.665005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.661666 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\") pod \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.699785 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": context deadline exceeded" Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.684795 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.699902 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.698221 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca" pod "6658141e-666b-4e5f-b6a3-ebb73346e0f3" (UID: "6658141e-666b-4e5f-b6a3-ebb73346e0f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.683776 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:56.780008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.719150 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:56.791008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.781256 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l" pod "6658141e-666b-4e5f-b6a3-ebb73346e0f3" (UID: "6658141e-666b-4e5f-b6a3-ebb73346e0f3"). InnerVolumeSpecName "kube-api-access-kn54l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:56.845009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.815509 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.877082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.872899 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca" pod "6658141e-666b-4e5f-b6a3-ebb73346e0f3" (UID: "6658141e-666b-4e5f-b6a3-ebb73346e0f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:56.877082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.820171 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:21:56.877082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.820853 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:21:56.877082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.820879 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:21:56.877082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.873275 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:21:56.930247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.825948 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\") pod \"9b565503-8053-4944-9cfc-a9a068ba8cb1\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " Nov 04 01:21:56.930247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.910871 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\" (UniqueName: \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\") pod \"9b565503-8053-4944-9cfc-a9a068ba8cb1\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " Nov 04 01:21:56.930247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.911047 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") pod \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\" (UID: \"6658141e-666b-4e5f-b6a3-ebb73346e0f3\") " Nov 04 01:21:56.930247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.911162 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\") pod \"9b565503-8053-4944-9cfc-a9a068ba8cb1\" (UID: \"9b565503-8053-4944-9cfc-a9a068ba8cb1\") " Nov 04 01:21:56.930247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.914826 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn54l\" (UniqueName: \"kubernetes.io/projected/6658141e-666b-4e5f-b6a3-ebb73346e0f3-kube-api-access-kn54l\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:56.934024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:56.820123 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.009013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.001050 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca" pod "9b565503-8053-4944-9cfc-a9a068ba8cb1" (UID: "9b565503-8053-4944-9cfc-a9a068ba8cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:57.101913 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:21:57.101882 4486 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/6658141e-666b-4e5f-b6a3-ebb73346e0f3/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 04 01:21:57.204246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.204148 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca" pod "6658141e-666b-4e5f-b6a3-ebb73346e0f3" (UID: "6658141e-666b-4e5f-b6a3-ebb73346e0f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:57.204463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.148495 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:57.217063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.217025 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.217326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.217292 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:21:57.242677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.242232 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:21:57.244366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.242949 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.257342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.257303 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.268198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.268162 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:57.333119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.328620 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.366789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.366747 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:57.432338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.432283 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca" pod "9b565503-8053-4944-9cfc-a9a068ba8cb1" (UID: "9b565503-8053-4944-9cfc-a9a068ba8cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:57.475127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.456854 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6658141e-666b-4e5f-b6a3-ebb73346e0f3-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:57.574470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.574414 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv" pod "9b565503-8053-4944-9cfc-a9a068ba8cb1" (UID: "9b565503-8053-4944-9cfc-a9a068ba8cb1"). InnerVolumeSpecName "kube-api-access-jf7kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:57.584009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.580417 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.589696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.589652 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:57.757038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.756867 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.758799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.758753 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:57.765005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.761089 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.798010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.773040 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:57.845007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.820523 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" Nov 04 01:21:57.849009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.844976 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:57.849009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.845828 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:57.898138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.880843 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.898138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.882105 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.898138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:57.883287 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.135161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.135108 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.284259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.284208 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.314122 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.285235 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.316776 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d" event={"ID":"6658141e-666b-4e5f-b6a3-ebb73346e0f3","Type":"ContainerDied","Data":"6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e"} Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.316840 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6adf5b7b04ee6ff61de83980c5b775085165d6f1d7efc2319a8365fe9fe8266e" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.316857 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc"] Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.317906 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9b565503-8053-4944-9cfc-a9a068ba8cb1" containerName="revoke-keys" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.317925 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b565503-8053-4944-9cfc-a9a068ba8cb1" containerName="revoke-keys" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.317954 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6658141e-666b-4e5f-b6a3-ebb73346e0f3" containerName="revoke-keys" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.317962 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6658141e-666b-4e5f-b6a3-ebb73346e0f3" containerName="revoke-keys" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.318144 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b565503-8053-4944-9cfc-a9a068ba8cb1" containerName="revoke-keys" Nov 04 01:21:58.329579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.318159 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6658141e-666b-4e5f-b6a3-ebb73346e0f3" containerName="revoke-keys" Nov 04 01:21:58.421299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.311455 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b565503-8053-4944-9cfc-a9a068ba8cb1-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:58.421541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.421517 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7kv\" (UniqueName: \"kubernetes.io/projected/9b565503-8053-4944-9cfc-a9a068ba8cb1-kube-api-access-jf7kv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:58.421631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.312086 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:21:58.431056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.312137 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:21:58.462135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.310958 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.578129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.578084 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:21:58.588097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.502162 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:21:58.618008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.502803 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Nov 04 01:21:58.618748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.526763 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" Nov 04 01:21:58.640039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.529565 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podStartSLOduration=8.529548132 podStartE2EDuration="8.529548132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:56.508196477 +0000 UTC m=+1485.266858640" watchObservedRunningTime="2026-11-04 01:21:58.529548132 +0000 UTC m=+1487.288210293" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.450388 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" podUID="fec97081-9922-47be-8eb7-3d25b8500e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.367342 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.674743 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.412397 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.674802 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.450078 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.674839 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.451417 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.674878 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.502744 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.674915 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.675159 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l" event={"ID":"9b565503-8053-4944-9cfc-a9a068ba8cb1","Type":"ContainerDied","Data":"4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03"} Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.675202 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4776f845b378ca355447a411153083504f739043e465a684782eab8899143e03" Nov 04 01:21:58.687605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.675221 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc"] Nov 04 01:21:58.741013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.725639 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:58.839082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.818409 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": context deadline exceeded" Nov 04 01:21:58.864871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.864818 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:58.895193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.882832 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:58.900890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.896287 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqfr\" (UniqueName: \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.019109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:58.998882 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:21:59.019109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.016210 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:21:59.019109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.016283 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:59.050217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.049086 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.050217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.049500 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.091328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.091297 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:21:59.108070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.106840 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqfr\" (UniqueName: \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.116534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.106851 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.135672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.135644 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:21:59.135923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.117087 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.165140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.128176 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.166038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.135575 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.166584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.116151 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7246-bae18f"] Nov 04 01:21:59.169591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.165438 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.192494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.192461 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.207208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.207091 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.207208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.207147 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:21:59.215017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.214195 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:21:59.222001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.218852 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqfr\" (UniqueName: \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\") pod \"maas-api-revoke-sub-d4416988-6v4hc\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.276078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.264277 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bqc6h\"" Nov 04 01:21:59.296013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.295739 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7246-bae18f"] Nov 04 01:21:59.308201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.308166 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.308444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.308416 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:21:59.312079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.309477 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.323097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.323063 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:21:59.323288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.323269 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:21:59.334036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.323712 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:21:59.359408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.350811 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.359408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.350862 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:21:59.511010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.443481 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:21:59.511582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.451013 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:21:59.511816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.511782 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:21:59.512587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.512557 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88d5\" (UniqueName: \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\") pod \"test-curl-7246-bae18f\" (UID: \"75bfaed5-1d0b-493d-8375-21c6593549e3\") " pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:21:59.613125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.552541 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:21:59.613125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.589160 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:59.694745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.692375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88d5\" (UniqueName: \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\") pod \"test-curl-7246-bae18f\" (UID: \"75bfaed5-1d0b-493d-8375-21c6593549e3\") " pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:21:59.921711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:21:59.910707 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.003912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.003069 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.083035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.060703 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.083035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.061146 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:00.112037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.098940 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.180193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.180058 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.237009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.224721 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.240135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.240093 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:00.248491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.248457 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.248712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.248686 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:00.266007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.250892 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.266283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.266251 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:00.292898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.292865 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.293184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.293153 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:00.414078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.357534 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88d5\" (UniqueName: \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\") pod \"test-curl-7246-bae18f\" (UID: \"75bfaed5-1d0b-493d-8375-21c6593549e3\") " pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:22:00.414078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.357631 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:00.414078 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.358003 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:00.428011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.416732 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:22:00.472074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.471963 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.531061 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:00.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.515113 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.505075 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:00.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.531507 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:00.620406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.620355 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:00.821064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:00.752530 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.030007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.016790 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.030007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.016910 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:01.076068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.046487 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.076068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.046598 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:01.105006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.076801 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.105006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.077229 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:01.121022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.108953 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.133008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.125883 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:01.205007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.202266 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.212006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.205543 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.212006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.205591 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:01.256523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.256486 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.256814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.256780 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:01.298043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.280642 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.298043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.280708 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:01.678086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.376292 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:01.680719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.415305 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.681262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.654113 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:01.686026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.683829 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.686026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.683885 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:01.729010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.683970 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.729544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.729505 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:01.729757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:01.695424 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.002953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.002859 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.003264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.003215 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:02.061116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.061065 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.083881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.083182 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.083881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.083581 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:02.129183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.129137 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:02.224090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.210170 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.224090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.211193 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.224090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.211236 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:02.268815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.268633 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.268815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.268694 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:02.280738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.280616 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.280738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.280672 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:02.384074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.383049 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.384074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.383109 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:02.461075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.458256 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.461075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.458318 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:02.608187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.608138 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6"] Nov 04 01:22:02.634009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.620293 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:02.733033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.708347 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:02.765323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.765271 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:02.766407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.766373 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:02.793750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.793700 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:02.793968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.793888 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:02.794148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.794108 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg2r\" (UniqueName: \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:02.798838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:02.798786 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.057066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.026643 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:03.057066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.026820 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:03.057066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.026953 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg2r\" (UniqueName: \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:03.057066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.028402 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:03.058691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.058659 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.058942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.058899 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:03.071007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.067315 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.095942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.095896 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.097483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.097453 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:03.114562 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:22:03.114524 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice/crio-a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7 WatchSource:0}: Error finding container a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7: Status 404 returned error can't find the container with id a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7 Nov 04 01:22:03.130195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.130160 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.416491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.212718 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.416926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.275953 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:03.417085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.315120 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.417228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.417199 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:03.417335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.373203 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.417449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.417431 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:03.463288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.463250 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.463530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.463503 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:03.463735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.463720 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.463866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.463850 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:03.471037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.466368 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.723446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.723353 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:03.743395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.743226 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:03.772243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.772195 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:03.806010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.790023 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:03.915010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:03.887053 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:04.124005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.121486 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:04.124005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.122335 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.124005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.122658 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.149006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.126887 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.149006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.126931 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:04.229862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.229825 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.231206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.231175 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:04.257011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.231623 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:22:04.290007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.289247 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.290007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.289300 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:04.373009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.363249 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.373330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.369622 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.390155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.390059 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:04.390408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.373289 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:04.417854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.417813 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.418640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.418599 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.469023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.461337 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.469023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.461396 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:04.469023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.463413 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg2r\" (UniqueName: \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\") pod \"maas-api-revoke-sub-8d71be16-g59p6\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:04.660461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.660426 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:04.661184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.661150 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Nov 04 01:22:04.789237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.789192 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.790943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.790909 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.815527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.815480 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:04.890260 4486 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.548s" Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.890305 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.890355 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6"] Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.890412 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.890436 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:22:04.893058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:04.890467 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc"] Nov 04 01:22:05.016007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.011199 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.016209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.016030 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7246-bae18f" event={"ID":"75bfaed5-1d0b-493d-8375-21c6593549e3","Type":"ContainerStarted","Data":"c1cf13e10980e10326e266d3bfd7d7c84af55a55ed0bf50fec73400ddb693d8d"} Nov 04 01:22:05.017814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.017785 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.018254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.018016 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:05.040016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.039962 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" event={"ID":"10a4fa9a-7b83-4040-ac51-a482ae0a5e89","Type":"ContainerStarted","Data":"a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7"} Nov 04 01:22:05.121970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.121926 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:05.122965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.122934 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.123373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.123350 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:05.211070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.211029 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.211463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.211339 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:05.283176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.283081 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.287006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.283393 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:05.291158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.291114 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.298093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.295597 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.298325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.298293 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:05.362218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.354133 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:22:05.362218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.355506 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.362218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.355558 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:05.469019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.468262 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:05.469019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.468747 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:05.545067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.503574 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.545067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.503647 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:05.741011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.720496 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:22:05.797113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.797019 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.798189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.797214 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.803194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.803168 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:22:05.803371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.803302 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:05.841009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.818018 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7246-bae18f"] Nov 04 01:22:05.841009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.818104 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.874475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.873753 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b45d1fe0-gw-openshift-default-dockercfg-9xwpp\"" Nov 04 01:22:05.882197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.882079 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:05.882197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.882147 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:05.983649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:05.959083 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.000774 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.000855 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.000921 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001017 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001246 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001337 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001398 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001598 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64cv\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.001678 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.011568 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.011889 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.012010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.011958 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:06.027250 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.027191 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.169007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.143074 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.257399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.148011 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.257876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.166959 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.258149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.258122 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.258325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.258304 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.258496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.258478 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.258790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.258768 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.263005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.259029 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.263222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.263199 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.288160 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.288116 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64cv\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.288473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.288451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.338011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.147774 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.369185 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.203692 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.369538 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.206487 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.468800 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.234140 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.249110 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.468929 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.338592 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.469007 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.470376 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.471550 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.471700 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.470054 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.492761 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.504369 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.513070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.505066 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.518620 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.518688 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:06.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.518862 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:06.539010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.518891 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.543012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.542950 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" event={"ID":"10a4fa9a-7b83-4040-ac51-a482ae0a5e89","Type":"ContainerStarted","Data":"68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e"} Nov 04 01:22:06.769062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.766732 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:06.866049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.815868 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:06.866049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.816276 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:07.018011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.989913 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:07.024010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:06.997817 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64cv\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:07.065214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.065164 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:07.139609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.139475 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.200765 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" podStartSLOduration=12.200738688 podStartE2EDuration="12.200738688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:06.996573812 +0000 UTC m=+1495.755235975" watchObservedRunningTime="2026-11-04 01:22:07.200738688 +0000 UTC m=+1495.959400850" Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.286583 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.286656 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.286774 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.286820 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.286959 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.310059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.287016 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:07.535089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.519947 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.535089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.520258 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.548007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.538318 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\") pod \"e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:07.579166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.579126 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:07.620142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.579455 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.580047 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.580165 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.631780 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.580214 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.631848 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.581601 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.642009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.631896 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:07.708132 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:22:07.707400 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081 WatchSource:0}: Error finding container 706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081: Status 404 returned error can't find the container with id 706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081 Nov 04 01:22:07.762009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.751599 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:07.953565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.953528 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" podUID="4bed0a38-3d09-483e-8c64-2bc858586a46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded" Nov 04 01:22:07.968555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.968503 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" podUID="c0d70473-bec3-4c76-95c2-7227ed44617b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:07.970634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.970603 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:07.971066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:07.971042 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:08.054039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.011824 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" podUID="fec97081-9922-47be-8eb7-3d25b8500e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:08.054039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.012219 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" podUID="344cadfc-a8d8-4ced-88d9-c046e7ecd3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.192448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.192337 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.276623 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.266161 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.320790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.298890 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.320790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.298950 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:08.320790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.299240 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.320790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.299270 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:08.387150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.387111 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" podUID="4bed0a38-3d09-483e-8c64-2bc858586a46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.394371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.394336 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.439307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.398060 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:08.453173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.449208 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.461066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.457064 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:08.465177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.453025 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:08.473007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.469129 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:08.529188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.529148 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6"] Nov 04 01:22:08.529509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.529485 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.529762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.529742 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.549301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.549254 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.549921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.549881 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.644016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.622811 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:08.644016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.623198 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:08.644016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.634078 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:08.674256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.674216 4486 generic.go:356] "Generic (PLEG): container finished" podID="10a4fa9a-7b83-4040-ac51-a482ae0a5e89" containerID="68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e" exitCode=0 Nov 04 01:22:08.678325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.678281 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" event={"ID":"10a4fa9a-7b83-4040-ac51-a482ae0a5e89","Type":"ContainerDied","Data":"68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e"} Nov 04 01:22:08.950005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.919071 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:22:09.060077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.975397 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.087090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.981944 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.087496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:08.991351 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.087657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.087626 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:09.087763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.016203 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.087954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.087935 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.088361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.088337 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.088529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.088511 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.088636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.088622 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.088783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.088770 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.088886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.088872 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.093006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.089023 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.093226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.093200 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.093579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.003247 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.106173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.031634 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:22:09.179010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.170521 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e400a3a6-openshift-default-dockercfg-t2v8k\"" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.197596 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.197666 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.197721 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.197791 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.198023 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.198083 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.198108 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.198170 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.198202 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.199113 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.230478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.199398 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.231916 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.231967 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.232478 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.233395 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.233644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:09.239031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.237194 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.284019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.277872 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.298871 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.278099 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.284374 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.300272 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.284849 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.338944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.300529 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:09.548828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.548787 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:09.549123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.549088 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:09.549670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.549634 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.637065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.611111 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:09.637065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.611510 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:09.637065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.611559 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.637065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.629202 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.637065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.629602 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:09.750107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.676106 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:09.784139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:09.775438 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.033344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.017119 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.033344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.017174 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:10.145679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.114430 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:10.233691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.160123 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:10.240437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.160165 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:10.277065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.195516 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.277065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.269598 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:10.277065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.203394 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:10.317578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.309904 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:10.331521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.331458 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.478052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.441246 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.478052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.441308 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:10.478052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.467673 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.478052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.467726 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:10.505156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.479591 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.505156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.479650 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:10.511576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.511532 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" event={"ID":"be675f53-aeb3-4c7e-9e0c-52b247f54ce3","Type":"ContainerStarted","Data":"706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081"} Nov 04 01:22:10.526772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.525109 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:10.526772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.525183 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:10.594066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.561227 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.611328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.611282 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:10.658141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.658095 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:10.659243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.659209 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:10.683562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.683518 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:10.702613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.702572 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:10.706113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.706067 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.059415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:10.770336 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": context deadline exceeded" Nov 04 01:22:11.066112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.050844 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:11.066340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.066309 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:11.066505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.051397 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.243056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.241653 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.279112 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.279639 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.279687 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.280297 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.282318 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:11.286027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.282358 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:11.338006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.319571 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.396021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.388683 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\") pod \"e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:11.420457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.418870 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:11.420457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.419028 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:11.453012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.438089 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:11.709029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.695103 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:11.709029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.695432 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:11.787959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.787923 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:11.789500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.788222 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:11.816668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.796552 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:11.816668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:11.804606 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:12.005489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.005453 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:12.005766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.005722 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:12.252977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.252847 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:12.280193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.280141 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:12.280463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.280426 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:12.302292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.302241 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:12.304901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.304874 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 04 01:22:12.305124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.305100 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 04 01:22:12.323272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.323236 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:12.469008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.458622 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:12.479005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.475302 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 04 01:22:12.483091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.483054 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 04 01:22:12.483277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.477815 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e400a3a6-openshift-default-dockercfg-t2v8k\"" Nov 04 01:22:12.487007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.463358 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:12.493041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.478608 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:12.494920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.477100 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 04 01:22:12.495125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.495096 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 04 01:22:12.539066 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:12.496965 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice/crio-conmon-68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice/crio-68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:12.753938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.607422 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:12.753938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.752971 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:12.753938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.753059 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:12.753938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.700042 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:12.754329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:12.700087 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:13.028032 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:13.023325 4486 info.go:109] Failed to get network devices: open /sys/class/net/a1b2aadf83010bc/address: no such file or directory Nov 04 01:22:13.036642 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:13.036598 4486 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.639s" Nov 04 01:22:13.040304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.040281 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:13.040490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.040471 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:22:13.145162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.145055 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.237476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.150081 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:13.347797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.331134 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.477881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.408401 4486 scope.go:122] "RemoveContainer" containerID="1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605" Nov 04 01:22:13.477881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.451963 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:13.477881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.452072 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.482174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.482132 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.500034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.489151 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.780487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.780374 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:13.781205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:13.781174 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.228023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.205206 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:22:14.228023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.210592 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.253956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.253912 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.259648 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerStarted","Data":"e43613d4a9f3413331dce6d04aff793dbf8321204b218d59dbb1f0d12bc900f6"} Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.259918 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.260184 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.260335 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.260452 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.260501 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:14.288113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.262246 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:14.301447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.301417 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:22:14.315008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.307811 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:14.315008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.312164 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.409009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.405647 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\") pod \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " Nov 04 01:22:14.409009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.405717 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\" (UniqueName: \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\") pod \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " Nov 04 01:22:14.409009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.405790 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\") pod \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\" (UID: \"10a4fa9a-7b83-4040-ac51-a482ae0a5e89\") " Nov 04 01:22:14.431018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.429088 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca" pod "10a4fa9a-7b83-4040-ac51-a482ae0a5e89" (UID: "10a4fa9a-7b83-4040-ac51-a482ae0a5e89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:14.499571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.439935 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:14.499571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.454997 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:14.499571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.455072 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.499571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.495243 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.533976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.533933 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:14.809022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.804309 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:14.809022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.804388 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:14.816255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.816224 4486 scope.go:122] "RemoveContainer" containerID="f1954d9f74033f2568e7fc1ad4c11be6486942a81932cfb28eb5abbf5caffc09" Nov 04 01:22:14.840402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.840360 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": context deadline exceeded" Nov 04 01:22:14.902006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.867404 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:15.106730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.907926 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:15.106950 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:14.961504 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.113578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.063101 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 04 01:22:15.113833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.113799 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 04 01:22:15.116034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.106254 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.274023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.261659 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:15.274023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.262962 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.329006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.312260 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:15.329006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.312614 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:15.329006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.312674 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.475058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.439830 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:15.475058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.457748 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.475058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.445403 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr" pod "10a4fa9a-7b83-4040-ac51-a482ae0a5e89" (UID: "10a4fa9a-7b83-4040-ac51-a482ae0a5e89"). InnerVolumeSpecName "kube-api-access-zrqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:15.587014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.576613 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": context deadline exceeded" Nov 04 01:22:15.588688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.588652 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.623380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.623336 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:15.623796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.623759 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:15.660717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.660654 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:15.661009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.660959 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.945119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.911114 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" podUID="c0d70473-bec3-4c76-95c2-7227ed44617b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.945119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.922216 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7" podUID="fec97081-9922-47be-8eb7-3d25b8500e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.985043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.961289 4486 scope.go:122] "RemoveContainer" containerID="4824b198356921b9ea6e0d8d0a53aff14efd66eeaabed2bad4b89642aa5eb108" Nov 04 01:22:15.985043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.961953 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" podUID="344cadfc-a8d8-4ced-88d9-c046e7ecd3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:15.985043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.980418 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:15.985043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.980475 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:16.009033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.985476 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8vwvq" podUID="4bed0a38-3d09-483e-8c64-2bc858586a46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 04 01:22:16.009033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:15.985396 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:16.192512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.157897 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:16.192512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.158377 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:16.192512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.158584 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:16.272537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.263630 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqfr\" (UniqueName: \"kubernetes.io/projected/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-kube-api-access-zrqfr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:16.278228 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:16.277171 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:22:16.278228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.277226 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:16.278228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.277463 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:16.278228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.277597 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": context deadline exceeded" Nov 04 01:22:16.362529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.362288 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:16.362716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.362571 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:16.362716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.362616 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:16.744007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.726112 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:16.744007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.726476 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:16.744007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.726737 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:16.768086 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:16.758665 4486 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.321s" Nov 04 01:22:16.768086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.762247 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:22:16.768086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.762324 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:22:16.768086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.762377 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:22:16.797304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.797215 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:22:16.834464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.834432 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:22:16.949005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.926817 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:22:16.962021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.952463 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" event={"ID":"10a4fa9a-7b83-4040-ac51-a482ae0a5e89","Type":"ContainerDied","Data":"a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7"} Nov 04 01:22:16.962021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.952532 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7" Nov 04 01:22:16.962021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:16.952854 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc" Nov 04 01:22:17.086094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.075130 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:17.086094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.075185 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.111147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.111103 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:17.111420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.111376 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.145499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.145446 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:17.145773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.145745 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:17.189005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.178971 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.204047 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.204089 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.176836 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca" pod "10a4fa9a-7b83-4040-ac51-a482ae0a5e89" (UID: "10a4fa9a-7b83-4040-ac51-a482ae0a5e89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.234599 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.236108 4486 generic.go:356] "Generic (PLEG): container finished" podID="be675f53-aeb3-4c7e-9e0c-52b247f54ce3" containerID="fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98" exitCode=0 Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.241331 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:22:17.282014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.279535 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" event={"ID":"be675f53-aeb3-4c7e-9e0c-52b247f54ce3","Type":"ContainerDied","Data":"fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98"} Nov 04 01:22:17.350020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.349126 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:22:17.419013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.412826 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.419013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.413743 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:22:17.419013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.413901 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:22:17.419013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.415414 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a4fa9a-7b83-4040-ac51-a482ae0a5e89-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:17.525025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.524624 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:17.525025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.524685 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534647 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534703 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534734 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534764 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534795 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534832 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:22:17.546084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.534876 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:17.643100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.641837 4486 scope.go:122] "RemoveContainer" containerID="81b07f3e6c049a9c33509af0575388adaf29d4c02334dc2d7a37bc37062f2fa5" Nov 04 01:22:17.658394 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:17.658362 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605\": container with ID starting with 1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605 not found: ID does not exist" containerID="1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605" Nov 04 01:22:17.659006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.658965 4486 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605\": container with ID starting with 1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605 not found: ID does not exist" Nov 04 01:22:17.675442 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:17.671013 4486 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605\": container with ID starting with 1fd7516acd3983440783ff41a162d9c8c58e20000d44d1fccfb2e1734b476605 not found: ID does not exist" Nov 04 01:22:17.713669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.713460 4486 scope.go:122] "RemoveContainer" containerID="a80512a36077f5b446ef4f990e5493f51bb24e6e113d61347d982f48f898ab83" Nov 04 01:22:17.742851 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:17.738263 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice/crio-a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:17.742851 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:17.739284 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a4fa9a_7b83_4040_ac51_a482ae0a5e89.slice/crio-a1b2aadf83010bc1396c4d133374167184abc49bd75f5a8f13b3cb7e25dacda7\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:17.766852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.764181 4486 scope.go:122] "RemoveContainer" containerID="215c28d55da41ea85c00252dd9b2f5963292606e1a09741a0bef99d782291d61" Nov 04 01:22:17.797959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.792837 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:22:17.798573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.798502 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:22:17.858051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.847972 4486 scope.go:122] "RemoveContainer" containerID="432b11040222a3b108bbd90a827935d712e5b0fc5a33d9692ca89c6ec4ed5a7f" Nov 04 01:22:17.996654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:17.995355 4486 scope.go:122] "RemoveContainer" containerID="0522dd12c2c122b40d27be1dee7985e46861026926e8d923cdf0db6098335856" Nov 04 01:22:18.014876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.014854 4486 scope.go:122] "RemoveContainer" containerID="018990968fb0353a52af27b6aa738dea7a87d757b8fc43841e3d48503064be78" Nov 04 01:22:18.065294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.065159 4486 scope.go:122] "RemoveContainer" containerID="fcd61f61f0408f7c8357bfb30ba0d5f0374f4624f7e75dcc4856c4048826039e" Nov 04 01:22:18.119355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.115097 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:18.119355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.115167 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:18.125073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.124947 4486 scope.go:122] "RemoveContainer" containerID="dd997a8a18c73ca8d8839384c2cd8e8c665989d1cf6fa23167da195913382f90" Nov 04 01:22:18.513807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.513711 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerStarted","Data":"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2"} Nov 04 01:22:18.513963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.513827 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerStarted","Data":"c5406801f6aa6486e19119c9d76ebd7b7df40a1cc8ffc45a9adf176a8280719d"} Nov 04 01:22:18.517137 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.517099 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerStarted","Data":"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff"} Nov 04 01:22:18.771556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.771465 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:18.780848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.780824 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:18.872591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.871431 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:18.980033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.976427 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\" (UniqueName: \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\") pod \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " Nov 04 01:22:18.980033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.976588 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\") pod \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " Nov 04 01:22:18.980033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.976644 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\") pod \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\" (UID: \"be675f53-aeb3-4c7e-9e0c-52b247f54ce3\") " Nov 04 01:22:18.987379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:18.987349 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca" pod "be675f53-aeb3-4c7e-9e0c-52b247f54ce3" (UID: "be675f53-aeb3-4c7e-9e0c-52b247f54ce3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:19.007900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.007861 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r" pod "be675f53-aeb3-4c7e-9e0c-52b247f54ce3" (UID: "be675f53-aeb3-4c7e-9e0c-52b247f54ce3"). InnerVolumeSpecName "kube-api-access-2jg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:19.033531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.033499 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca" pod "be675f53-aeb3-4c7e-9e0c-52b247f54ce3" (UID: "be675f53-aeb3-4c7e-9e0c-52b247f54ce3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:19.088799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.088761 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jg2r\" (UniqueName: \"kubernetes.io/projected/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-kube-api-access-2jg2r\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:19.089216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.089193 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:19.089336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.089322 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be675f53-aeb3-4c7e-9e0c-52b247f54ce3-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:19.155169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.155124 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:19.155472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.155439 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:19.181681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.181647 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:22:19.183587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.183524 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:22:19.460043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.459439 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podStartSLOduration=16.459422669 podStartE2EDuration="16.459422669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:19.443692154 +0000 UTC m=+1508.202354315" watchObservedRunningTime="2026-11-04 01:22:19.459422669 +0000 UTC m=+1508.218084829" Nov 04 01:22:19.479155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.479125 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:19.487442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.487384 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:19.528809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.528781 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" Nov 04 01:22:19.528961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.528838 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6" event={"ID":"be675f53-aeb3-4c7e-9e0c-52b247f54ce3","Type":"ContainerDied","Data":"706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081"} Nov 04 01:22:19.528961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.528891 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706bfc3dfeafab0d26ba5b13b63640b01563fe614015a185ebb2af083e4a1081" Nov 04 01:22:19.529776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.529754 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:19.529897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.529801 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:19.530598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.530578 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:22:19.530675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.530628 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:22:19.579525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:19.579322 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podStartSLOduration=12.579300797 podStartE2EDuration="12.579300797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:19.567553954 +0000 UTC m=+1508.326216113" watchObservedRunningTime="2026-11-04 01:22:19.579300797 +0000 UTC m=+1508.337962971" Nov 04 01:22:21.336927 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:21.336894 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:21.554455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:21.554417 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7246-bae18f" event={"ID":"75bfaed5-1d0b-493d-8375-21c6593549e3","Type":"ContainerStarted","Data":"b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947"} Nov 04 01:22:21.664246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:21.664143 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7246-bae18f" podStartSLOduration=9.049911986 podStartE2EDuration="25.664129162s" totalImagesPullingTime="16.614217176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:21.663920108 +0000 UTC m=+1510.422582271" watchObservedRunningTime="2026-11-04 01:22:21.664129162 +0000 UTC m=+1510.422791323" Nov 04 01:22:23.122738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:23.122688 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7246-bae18f"] Nov 04 01:22:23.129316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:23.129288 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7246-bae18f"] Nov 04 01:22:23.577767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:23.577699 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7246-bae18f" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" containerName="test-curl-7246-bae18f" containerID="cri-o://b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947" gracePeriod=2 Nov 04 01:22:25.858465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:25.858439 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:22:25.915873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:25.915837 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\" (UniqueName: \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\") pod \"75bfaed5-1d0b-493d-8375-21c6593549e3\" (UID: \"75bfaed5-1d0b-493d-8375-21c6593549e3\") " Nov 04 01:22:25.917999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:25.917962 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5" pod "75bfaed5-1d0b-493d-8375-21c6593549e3" (UID: "75bfaed5-1d0b-493d-8375-21c6593549e3"). InnerVolumeSpecName "kube-api-access-m88d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:26.018335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.018300 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m88d5\" (UniqueName: \"kubernetes.io/projected/75bfaed5-1d0b-493d-8375-21c6593549e3-kube-api-access-m88d5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:26.610373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.610340 4486 generic.go:356] "Generic (PLEG): container finished" podID="75bfaed5-1d0b-493d-8375-21c6593549e3" containerID="b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947" exitCode=137 Nov 04 01:22:26.610556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.610381 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:22:26.610556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.610445 4486 scope.go:122] "RemoveContainer" containerID="b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947" Nov 04 01:22:26.622831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.622813 4486 scope.go:122] "RemoveContainer" containerID="b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947" Nov 04 01:22:26.623097 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:26.623077 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947\": container with ID starting with b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947 not found: ID does not exist" containerID="b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947" Nov 04 01:22:26.623156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:26.623106 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947"} err="failed to get container status \"b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947\": rpc error: code = NotFound desc = could not find container \"b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947\": container with ID starting with b2ec1e7f2c9dd375bc53380775abf20ee397662eb73816416503311e15179947 not found: ID does not exist" Nov 04 01:22:26.670133 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:26.670096 4486 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7246-bae18f\" is forbidden: User \"system:node:ip-10-0-129-114.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-114.us-west-2.compute.internal' and this object" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" pod="opendatahub/test-curl-7246-bae18f" Nov 04 01:22:26.894444 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:26.894352 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:27.352093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:27.352043 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" path="/var/lib/kubelet/pods/75bfaed5-1d0b-493d-8375-21c6593549e3/volumes" Nov 04 01:22:29.738993 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.738951 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s"] Nov 04 01:22:29.740001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.739965 4486 cpu_manager.go:414] "containerMap: removing container" podUID="be675f53-aeb3-4c7e-9e0c-52b247f54ce3" containerName="revoke-keys" Nov 04 01:22:29.740179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740153 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be675f53-aeb3-4c7e-9e0c-52b247f54ce3" containerName="revoke-keys" Nov 04 01:22:29.740275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740193 4486 cpu_manager.go:414] "containerMap: removing container" podUID="10a4fa9a-7b83-4040-ac51-a482ae0a5e89" containerName="revoke-keys" Nov 04 01:22:29.740275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740203 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a4fa9a-7b83-4040-ac51-a482ae0a5e89" containerName="revoke-keys" Nov 04 01:22:29.740275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740213 4486 cpu_manager.go:414] "containerMap: removing container" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" containerName="test-curl-7246-bae18f" Nov 04 01:22:29.740275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740221 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" containerName="test-curl-7246-bae18f" Nov 04 01:22:29.740512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740484 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="be675f53-aeb3-4c7e-9e0c-52b247f54ce3" containerName="revoke-keys" Nov 04 01:22:29.740512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740504 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a4fa9a-7b83-4040-ac51-a482ae0a5e89" containerName="revoke-keys" Nov 04 01:22:29.740616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.740516 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bfaed5-1d0b-493d-8375-21c6593549e3" containerName="test-curl-7246-bae18f" Nov 04 01:22:29.746387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.746364 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:29.917015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.905316 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:22:29.917404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.917372 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:29.974749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.974713 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s"] Nov 04 01:22:30.005249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:29.976699 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.005633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.005597 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.005901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.005872 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2gn\" (UniqueName: \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.107775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.107738 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.107951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.107836 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2gn\" (UniqueName: \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.108171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.108150 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.108769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.108748 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.110115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.110093 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.163431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.163404 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2gn\" (UniqueName: \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\") pod \"maas-api-revoke-sub-c85061fb-gsv4s\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.363069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.363034 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:30.741283 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:22:30.741252 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef22b75_b5bb_4ea0_9680_e49aa3e2a8d2.slice/crio-ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148 WatchSource:0}: Error finding container ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148: Status 404 returned error can't find the container with id ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148 Nov 04 01:22:30.742210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:30.742180 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s"] Nov 04 01:22:31.679002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:31.678952 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" event={"ID":"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2","Type":"ContainerStarted","Data":"820e3b81ac63b0fb586436936ce130eb4190a9819bb940f213dbccfdf06d40f7"} Nov 04 01:22:31.679180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:31.679050 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" event={"ID":"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2","Type":"ContainerStarted","Data":"ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148"} Nov 04 01:22:31.793124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:31.793071 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" podStartSLOduration=2.793055665 podStartE2EDuration="2.793055665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:31.787834365 +0000 UTC m=+1520.546496531" watchObservedRunningTime="2026-11-04 01:22:31.793055665 +0000 UTC m=+1520.551717825" Nov 04 01:22:32.690249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:32.690213 4486 generic.go:356] "Generic (PLEG): container finished" podID="bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" containerID="820e3b81ac63b0fb586436936ce130eb4190a9819bb940f213dbccfdf06d40f7" exitCode=0 Nov 04 01:22:32.690429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:32.690302 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" event={"ID":"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2","Type":"ContainerDied","Data":"820e3b81ac63b0fb586436936ce130eb4190a9819bb940f213dbccfdf06d40f7"} Nov 04 01:22:33.875402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.875375 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:33.971522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.971430 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\") pod \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " Nov 04 01:22:33.971702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.971528 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\") pod \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " Nov 04 01:22:33.971702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.971556 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\" (UniqueName: \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\") pod \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\" (UID: \"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2\") " Nov 04 01:22:33.971928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.971903 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca" pod "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" (UID: "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:33.972327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.972302 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:33.973978 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.973944 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca" pod "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" (UID: "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:33.974379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:33.974355 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn" pod "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" (UID: "bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2"). InnerVolumeSpecName "kube-api-access-tb2gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:34.073528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.073492 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:34.073528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.073523 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2gn\" (UniqueName: \"kubernetes.io/projected/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2-kube-api-access-tb2gn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:34.724039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.723369 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" event={"ID":"bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2","Type":"ContainerDied","Data":"ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148"} Nov 04 01:22:34.724039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.723438 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ede1fa7a613b8a5f021bf344158b48ec0b2e6c79e2061e9f7fa271334bc95148" Nov 04 01:22:34.724039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.723505 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s" Nov 04 01:22:34.990749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.990662 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:22:34.991387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.991370 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" containerName="revoke-keys" Nov 04 01:22:34.991459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.991388 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" containerName="revoke-keys" Nov 04 01:22:34.991541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.991531 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" containerName="revoke-keys" Nov 04 01:22:35.004006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:34.996223 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.120472 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.120575 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.120630 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.120894 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.120939 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.121046 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hhm\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.121112 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.125151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.121595 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.125151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.121690 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.152514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.152442 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262405 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262526 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262600 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262651 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262758 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262786 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262817 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hhm\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.262877 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.263189 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.264484 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.264735 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.264963 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.265235 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.287061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.265937 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.343233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.313284 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.350656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.348940 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.400069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.387251 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.400069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.387957 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hhm\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\") pod \"e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.622262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.622226 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:35.722534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.720973 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:22:35.728524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.726067 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.778783 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.778835 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.778888 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.778953 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.779142 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.779207 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.779247 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.779287 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j4nd\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.779510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.779418 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.790008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.781715 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:22:35.880996 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.880882 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881000 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881050 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881093 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j4nd\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881235 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881270 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881301 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881367 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.881599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.881435 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.882085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.882056 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.883298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.883268 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.883569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.883545 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.884263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.884233 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.884596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.884548 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.886241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.886214 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.887312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.887286 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.938637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.938602 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:35.938775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:35.938607 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j4nd\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\") pod \"e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:36.070191 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:22:36.070148 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a8f3ad_f89b_4079_98b7_df4fc1b156e1.slice/crio-21ca26f04603c2fe5e912d92d692aa0f758dbf8726f395a5d1736042035e4cbe WatchSource:0}: Error finding container 21ca26f04603c2fe5e912d92d692aa0f758dbf8726f395a5d1736042035e4cbe: Status 404 returned error can't find the container with id 21ca26f04603c2fe5e912d92d692aa0f758dbf8726f395a5d1736042035e4cbe Nov 04 01:22:36.071359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.071334 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:22:36.072946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.072911 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.073145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.073006 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.073145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.073052 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.082333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.082303 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:36.410165 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:36.410075 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:36.589226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.582816 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:22:36.593185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.593148 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.593426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.593399 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.593567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.593551 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:36.773096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.770199 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" event={"ID":"af563074-bd2d-4b27-a57d-0a1f9d26b186","Type":"ContainerStarted","Data":"ff270a2213068a1f67c1dc31f30cf94d0d20720bf64dfccc40b099b51c191e30"} Nov 04 01:22:36.776084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.774602 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" event={"ID":"46a8f3ad-f89b-4079-98b7-df4fc1b156e1","Type":"ContainerStarted","Data":"295e505ebc89a9cacf37b0987d533b4f883f76fb1542a4bb6c556507822eeb19"} Nov 04 01:22:36.776084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.774662 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" event={"ID":"46a8f3ad-f89b-4079-98b7-df4fc1b156e1","Type":"ContainerStarted","Data":"21ca26f04603c2fe5e912d92d692aa0f758dbf8726f395a5d1736042035e4cbe"} Nov 04 01:22:36.917945 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:36.911658 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podStartSLOduration=2.911637105 podStartE2EDuration="2.911637105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:36.895246764 +0000 UTC m=+1525.653908926" watchObservedRunningTime="2026-11-04 01:22:36.911637105 +0000 UTC m=+1525.670299267" Nov 04 01:22:37.060110 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:37.047020 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:37.631836 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:37.631800 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:37.643053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:37.635120 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:37.665188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:37.643384 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:38.113011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.105115 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.128017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.108240 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.231387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.229311 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.338300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.252084 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.431672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.431328 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Nov 04 01:22:38.513425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.462952 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.550302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.550242 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.578942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.568440 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:38.630046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.627027 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:22:38.728199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.728097 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:38.792169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.792130 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:38.792468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.792436 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:38.857297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.857263 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891545 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6cd\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891687 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891720 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891782 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891930 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.891969 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.936004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.933670 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:38.993052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:38.973564 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:22:39.017012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.012306 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:22:39.022153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.022120 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:22:39.050056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.026189 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7bjdx\"" Nov 04 01:22:39.050256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.039722 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.066010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.050527 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.146535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.146501 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.162590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.146748 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.163049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.163018 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6cd\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.163343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.163326 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.163467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.163451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.172021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.100491 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:22:39.172201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.128762 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.172848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.140160 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.202355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.146403 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.211006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.152781 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.211244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.176164 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:22:39.211446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.211427 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:22:39.219763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.219736 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" event={"ID":"af563074-bd2d-4b27-a57d-0a1f9d26b186","Type":"ContainerStarted","Data":"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a"} Nov 04 01:22:39.220638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.216920 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.224074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.217389 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.224223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.176924 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:39.224371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.224345 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.258141 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.258536 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.258839 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.258898 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.277297 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.378877 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6cd\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.435065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.403638 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fn8kh\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:39.572007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.562912 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.581205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.581160 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.593928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.583870 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.629896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.568061 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.629896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.610655 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:22:39.629896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.610721 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:22:39.629896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.610758 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:22:39.812106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.730094 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:39.812106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.766865 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.812106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.805767 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:39.812106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.805831 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:39.871385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.837064 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:39.987015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.955078 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:40.001052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.987268 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:40.035003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:39.966089 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:40.160008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.135504 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:40.164005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.159461 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:40.164005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.161145 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:40.210010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.204333 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.235162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.235117 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:40.386487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.280088 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.456402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.280142 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.586032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.584739 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.613014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.612150 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:40.613014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.612524 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:40.613014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.612833 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.613283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.613134 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:22:40.714064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.708145 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:40.714064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.713146 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podStartSLOduration=5.713125869 podStartE2EDuration="5.713125869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:39.660533836 +0000 UTC m=+1528.419196009" watchObservedRunningTime="2026-11-04 01:22:40.713125869 +0000 UTC m=+1529.471788032" Nov 04 01:22:40.727112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.727045 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:40.793170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.793106 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:40.862110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.862065 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8" podUID="c0d70473-bec3-4c76-95c2-7227ed44617b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.874608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.874564 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zrnhb" podUID="344cadfc-a8d8-4ced-88d9-c046e7ecd3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:40.875184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:40.875155 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": context deadline exceeded" Nov 04 01:22:41.041226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.017207 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": context deadline exceeded" Nov 04 01:22:41.207548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.142234 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:41.207548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.142298 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:41.237050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.211809 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.341462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.324081 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:41.341462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.324145 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.467786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.392175 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.467786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.410773 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:41.467786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.451868 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.590006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.589081 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:41.649227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.646791 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:41.649227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.647215 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:41.649227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.647506 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:41.675466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.675429 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:41.694192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.694148 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:41.805319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.805273 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:41.961064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.957224 4486 patch_prober.go:29] interesting pod/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:41.961064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:41.957310 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.244072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.225261 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:42.244072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.225323 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:42.245425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.245385 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.494010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.493098 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:42.494010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.493450 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:42.494310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.494011 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:42.680796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.662408 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.699403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.699341 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:42.701112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.700968 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:42.721868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.720107 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:42.782125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.766599 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:42.782125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.766672 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:42.811277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.811222 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:42.973037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.961146 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:42.973037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:42.961212 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.084142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.084109 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:43.084425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.084380 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:43.292038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.275489 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:43.338213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.338112 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:43.338481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.338440 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.516033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.500616 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:43.516668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.500846 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:43.529368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.500877 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:43.664073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.661279 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:43.664073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.661357 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:43.674113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.674065 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:43.762192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.741456 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:43.822177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.778575 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:43.822755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.778621 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:43.823159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:43.819515 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.103025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.081230 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:44.103025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.081290 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.176971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.155192 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.176971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.155279 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:44.178181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.178145 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:44.178357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.178330 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.275937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.275794 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.474342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.474247 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:22:44.526169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.521654 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.526169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.522157 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 04 01:22:44.542036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.537067 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.609013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.606124 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:44.609013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.606223 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.620823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.617076 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:44.626658 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:44.626621 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:22:44.639954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.639921 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:44.640247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.640211 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:44.695190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.695144 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:44.872074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.836089 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:44.872074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.855155 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:44.872074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.855513 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:44.902406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:44.902358 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": context deadline exceeded" Nov 04 01:22:45.154065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.128144 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.154065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.147428 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:45.301038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.300068 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.301038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.300824 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.397009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.387914 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:45.397009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.387997 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.524070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.522252 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.524070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.523208 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.538633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.538591 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:45.637863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.637814 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.641093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.641054 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:45.726120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.726077 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.846030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.845962 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.894563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.894517 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:45.909307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.902583 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:45.964252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:45.948220 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.084010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.075626 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" event={"ID":"2111fa87-0b9d-47cf-b1de-f5406ca8fd18","Type":"ContainerStarted","Data":"9ebb5f6193d92ea245269afa59fdcf473e3e8cb763fc55fc58e0ac011f4041dc"} Nov 04 01:22:46.116504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.115209 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.128295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.128163 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:46.161256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.161197 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.306657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.306610 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.308929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.308893 4486 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.525443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.525317 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": context deadline exceeded" Nov 04 01:22:46.534186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.534129 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.569036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.548701 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:46.623366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.623330 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.625109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.625066 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:46.754128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.754078 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": context deadline exceeded" Nov 04 01:22:46.904118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.904061 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.909412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.909371 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:46.932891 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:46.932812 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.006653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.003524 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:22:47.094023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.093281 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.094023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.093338 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:47.162760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.162324 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.308850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.308777 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.544866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.544731 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.591111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.591080 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:22:47.614977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.591659 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:22:47.658691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.649864 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.658691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.649918 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:47.898077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.873079 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.914466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.914421 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Nov 04 01:22:47.967006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:47.966073 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.160336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.136147 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded" Nov 04 01:22:48.199008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.198085 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.272006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.269681 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.272006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.269784 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:48.297010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.296770 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:22:48.316089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.315576 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" probeResult="failure" output="Get \"https://10.133.0.115:8000/ready\": context deadline exceeded" Nov 04 01:22:48.421011 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:48.420887 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:48.504448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.491378 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:22:48.507215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.504896 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:22:48.507215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.505829 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:22:48.507215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.506629 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:22:48.507215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.506839 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:22:48.510839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.510816 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:22:48.626970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.624151 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.626970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:48.624208 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:49.087013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:49.086409 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.087013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:49.086470 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:49.622943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:49.622907 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.623480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:49.622964 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:50.092007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:50.089069 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.092007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:50.089130 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:50.624223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:50.623073 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:50.624223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:50.623968 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.624223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:50.624022 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:51.099551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:51.097963 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:51.099551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:51.099111 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.099551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:51.099161 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:51.319002 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:51.315819 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:51.623717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:51.623362 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.623717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:51.623422 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Nov 04 01:22:52.083493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:52.083455 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.084117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:52.084060 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Nov 04 01:22:53.304211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.304178 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:53.307320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.307291 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:22:53.666187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.666066 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:53.666187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.666133 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.908147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.908106 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:53.908301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:53.908210 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:22:54.287446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:54.287357 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:54.288928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:54.288902 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:22:54.624298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:54.624193 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" event={"ID":"2111fa87-0b9d-47cf-b1de-f5406ca8fd18","Type":"ContainerStarted","Data":"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918"} Nov 04 01:22:54.752611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:54.752536 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" podStartSLOduration=7.920285884 podStartE2EDuration="17.752516572s" totalImagesPullingTime="9.832230688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:54.750317253 +0000 UTC m=+1543.508979416" watchObservedRunningTime="2026-11-04 01:22:54.752516572 +0000 UTC m=+1543.511178735" Nov 04 01:22:55.643884 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:55.643214 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:55.654783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:55.654663 4486 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fn8kh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:22:55.654930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:55.654767 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:22:56.656628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:22:56.656591 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:22:58.545171 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:22:58.545130 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:06.378326 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:23:06.378287 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:08.565279 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:23:08.565239 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-conmon-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe675f53_aeb3_4c7e_9e0c_52b247f54ce3.slice/crio-fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:13.857745 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:23:13.857703 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:23:18.412218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:18.412185 4486 scope.go:122] "RemoveContainer" containerID="b7d2cab5bbbd76a7595d08eaa88cf8ffa5e13a3c2316db61e3ab865a76b734bb" Nov 04 01:23:18.424327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:18.424306 4486 scope.go:122] "RemoveContainer" containerID="5b74dea08064480f2a887ab41f90fc96042bc8c801dd2f14906c91efe8e0ecd2" Nov 04 01:23:18.434617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:18.434591 4486 scope.go:122] "RemoveContainer" containerID="52438940aed9c9ace7258a71b3bed1d5db66bd950f1bc17063c2064ee7004705" Nov 04 01:23:18.444821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:18.444795 4486 scope.go:122] "RemoveContainer" containerID="e0bb871842c2fd20214bb0d080494a7f6d48f7204ac95116a8235746f2621eb4" Nov 04 01:23:27.320234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.320159 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq"] Nov 04 01:23:27.326864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.326840 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.370001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.369943 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l"] Nov 04 01:23:27.374224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.374208 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.377498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.377473 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fwrps\"" Nov 04 01:23:27.418496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.418464 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq"] Nov 04 01:23:27.420439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420408 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd2b86c-c6f6-4156-ba26-5b1116526464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420487 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420513 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420595 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420651 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420741 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.420940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420914 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.421054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.420955 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlf7c\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-kube-api-access-wlf7c\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.421184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.421161 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.486293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.486259 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l"] Nov 04 01:23:27.523208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523178 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523309 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.523397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523338 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.523397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd2b86c-c6f6-4156-ba26-5b1116526464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523576 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523616 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523687 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a49baee-3767-4b1b-b167-da380781cbaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.523741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523715 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523742 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523789 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523827 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.523898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523859 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.523898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523877 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.524053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523909 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a49baee-3767-4b1b-b167-da380781cbaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.524053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.523931 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxb5d\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-kube-api-access-wxb5d\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.524138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524066 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.524138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.524209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524152 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.524209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524187 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlf7c\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-kube-api-access-wlf7c\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.524326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524286 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.524326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524301 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.524542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524513 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd2b86c-c6f6-4156-ba26-5b1116526464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.524720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.524697 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.525773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.525751 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.526618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.526599 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.581680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.581606 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlf7c\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-kube-api-access-wlf7c\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.581843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.581694 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd2b86c-c6f6-4156-ba26-5b1116526464-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4wlbq\" (UID: \"8dd2b86c-c6f6-4156-ba26-5b1116526464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.625471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625440 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a49baee-3767-4b1b-b167-da380781cbaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625508 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625563 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625591 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a49baee-3767-4b1b-b167-da380781cbaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625614 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxb5d\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-kube-api-access-wxb5d\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625651 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625696 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.625953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625918 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625967 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.626250 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.625919 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.626281 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.626338 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.626563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.626373 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a49baee-3767-4b1b-b167-da380781cbaa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.627901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.627879 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a49baee-3767-4b1b-b167-da380781cbaa-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.628030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.627883 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a49baee-3767-4b1b-b167-da380781cbaa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.651266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.651236 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:27.680774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.680745 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.681255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.681235 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxb5d\" (UniqueName: \"kubernetes.io/projected/2a49baee-3767-4b1b-b167-da380781cbaa-kube-api-access-wxb5d\") pod \"maas-default-gateway-openshift-default-694db55c87-cdh5l\" (UID: \"2a49baee-3767-4b1b-b167-da380781cbaa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:27.688069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:27.688050 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:28.158540 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:23:28.158496 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd2b86c_c6f6_4156_ba26_5b1116526464.slice/crio-74f34cb1dc11ebb724cc4ce41522e40a5f7cf44a9445fedcc770dfcbfe74fe6c WatchSource:0}: Error finding container 74f34cb1dc11ebb724cc4ce41522e40a5f7cf44a9445fedcc770dfcbfe74fe6c: Status 404 returned error can't find the container with id 74f34cb1dc11ebb724cc4ce41522e40a5f7cf44a9445fedcc770dfcbfe74fe6c Nov 04 01:23:28.161585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.161555 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq"] Nov 04 01:23:28.161701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.161575 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.161701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.161691 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.161824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.161735 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.165380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.165348 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.165578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.165550 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.165692 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.165603 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:28.168334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:28.168308 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l"] Nov 04 01:23:29.110419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.110376 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" event={"ID":"8dd2b86c-c6f6-4156-ba26-5b1116526464","Type":"ContainerStarted","Data":"0e4ac8cfe2ded7632f0869cba18505fe7152673ef190365d3bab8b81c9ee0c8b"} Nov 04 01:23:29.110912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.110465 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" event={"ID":"8dd2b86c-c6f6-4156-ba26-5b1116526464","Type":"ContainerStarted","Data":"74f34cb1dc11ebb724cc4ce41522e40a5f7cf44a9445fedcc770dfcbfe74fe6c"} Nov 04 01:23:29.112289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.112260 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" event={"ID":"2a49baee-3767-4b1b-b167-da380781cbaa","Type":"ContainerStarted","Data":"f4c8f848ba8504f48668b73be105c6bfb1c1922b7bba6df02d80510a7a8486aa"} Nov 04 01:23:29.112414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.112297 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" event={"ID":"2a49baee-3767-4b1b-b167-da380781cbaa","Type":"ContainerStarted","Data":"bd9a41a46023a1a8584c7f37840bf3e0a1d009e2e70893323b0805aa7f7fe031"} Nov 04 01:23:29.293354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.282949 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podStartSLOduration=2.282930667 podStartE2EDuration="2.282930667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:29.280459666 +0000 UTC m=+1578.039121826" watchObservedRunningTime="2026-11-04 01:23:29.282930667 +0000 UTC m=+1578.041592829" Nov 04 01:23:29.405801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.405691 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podStartSLOduration=2.405676064 podStartE2EDuration="2.405676064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:29.396958954 +0000 UTC m=+1578.155621125" watchObservedRunningTime="2026-11-04 01:23:29.405676064 +0000 UTC m=+1578.164338298" Nov 04 01:23:29.656072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.655086 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:29.656072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.655249 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:29.656072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.655307 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:29.690667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.690620 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:29.691053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.691018 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:29.691219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:29.691197 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:30.675009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:30.671168 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:30.675009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:30.671239 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:30.695006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:30.693326 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:30.695006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:30.693393 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:31.655005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:31.654264 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.655005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:31.654334 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:31.692472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:31.692438 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.693088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:31.693057 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:32.654400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:32.654361 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:32.654727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:32.654693 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:32.693716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:32.693680 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:32.694293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:32.694258 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:33.654008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:33.653286 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:33.654008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:33.653350 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:33.690266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:33.689377 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:33.690266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:33.689438 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:34.563688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.563641 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.622161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.622112 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.652165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.652108 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.654026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.653977 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.654300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.654276 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:34.689648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.689611 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.689932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:34.689900 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:35.669669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:35.666080 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.669669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:35.666153 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:35.688874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:35.688841 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.689148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:35.689116 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:36.653007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:36.652280 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.653007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:36.652375 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 04 01:23:36.689325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:36.689285 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.689810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:36.689333 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 04 01:23:38.624066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.623958 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:38.625873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.625848 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:38.626794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.626772 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" Nov 04 01:23:38.694008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.692561 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:38.694008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.692672 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:38.816007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.815170 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:38.816007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.815857 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:38.818970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:38.816641 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" Nov 04 01:23:42.457515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.457481 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn"] Nov 04 01:23:42.483226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.483196 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.588969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.588933 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn"] Nov 04 01:23:42.614802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.614758 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615022 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.614952 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615213 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7f914-cc9b-4ead-9299-822ce680bc33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615267 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615392 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615584 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2d6\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-kube-api-access-zr2d6\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615788 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.615898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.615825 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.716992 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.716895 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717055 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2d6\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-kube-api-access-zr2d6\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717086 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717103 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717139 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717254 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717310 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717344 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717453 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717461 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7f914-cc9b-4ead-9299-822ce680bc33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717508 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717655 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.717795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717775 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.718026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.717998 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc7f914-cc9b-4ead-9299-822ce680bc33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.719593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.719570 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.719975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.719949 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.772431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.772396 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2d6\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-kube-api-access-zr2d6\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.773852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.773821 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc7f914-cc9b-4ead-9299-822ce680bc33-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-dkxqn\" (UID: \"4fc7f914-cc9b-4ead-9299-822ce680bc33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:42.806490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:42.806461 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:43.310430 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:23:43.310395 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc7f914_cc9b_4ead_9299_822ce680bc33.slice/crio-b95bc716af5ec02bc0973f0875125139ab3177bc20440fb922b7558d99f76ac1 WatchSource:0}: Error finding container b95bc716af5ec02bc0973f0875125139ab3177bc20440fb922b7558d99f76ac1: Status 404 returned error can't find the container with id b95bc716af5ec02bc0973f0875125139ab3177bc20440fb922b7558d99f76ac1 Nov 04 01:23:43.312248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.312163 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn"] Nov 04 01:23:43.313264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.313230 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:43.313376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.313296 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:43.313376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.313323 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:43.452209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.452023 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" event={"ID":"4fc7f914-cc9b-4ead-9299-822ce680bc33","Type":"ContainerStarted","Data":"f4c4b665de1d9ff74da0c33e12f5ec39885880bd42b25a2472e181c14297b47b"} Nov 04 01:23:43.452397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.452244 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" event={"ID":"4fc7f914-cc9b-4ead-9299-822ce680bc33","Type":"ContainerStarted","Data":"b95bc716af5ec02bc0973f0875125139ab3177bc20440fb922b7558d99f76ac1"} Nov 04 01:23:43.569837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.569779 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podStartSLOduration=1.569762573 podStartE2EDuration="1.569762573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:43.566049076 +0000 UTC m=+1592.324711237" watchObservedRunningTime="2026-11-04 01:23:43.569762573 +0000 UTC m=+1592.328424734" Nov 04 01:23:43.806663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:43.806619 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:43.845808 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:23:43.845705 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:23:44.474483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.474397 4486 generic.go:356] "Generic (PLEG): container finished" podID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerID="215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2" exitCode=0 Nov 04 01:23:44.474717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.474475 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerDied","Data":"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2"} Nov 04 01:23:44.475135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.475114 4486 scope.go:122] "RemoveContainer" containerID="215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2" Nov 04 01:23:44.476654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.476625 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:44.476738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.476687 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:44.476738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.476719 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:23:44.811695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.808958 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:44.811695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:44.809092 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:45.486846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:45.486803 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerStarted","Data":"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37"} Nov 04 01:23:45.806879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:45.806786 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.806879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:45.806837 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:46.480420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:46.480382 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:23:46.481143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:46.481118 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.481318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:46.481294 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 01:23:46.807414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:46.807312 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.807414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:46.807367 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:47.492009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:47.487430 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 01:23:47.492009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:47.487486 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 01:23:47.807155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:47.807059 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:47.807301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:47.807133 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:48.484008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:48.482969 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 01:23:48.484008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:48.483059 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 01:23:48.807366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:48.807285 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:48.807366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:48.807336 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:49.486008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:49.485647 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 01:23:49.486008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:49.485707 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 01:23:49.536009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:49.530755 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:23:49.807482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:49.807391 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:49.807482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:49.807458 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:50.479606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:50.479566 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 04 01:23:50.479797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:50.479633 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 04 01:23:50.806864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:50.806768 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-dkxqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 04 01:23:50.806864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:50.806823 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" podUID="4fc7f914-cc9b-4ead-9299-822ce680bc33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 04 01:23:51.484343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:51.484311 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:23:51.485455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:51.485431 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:23:52.541221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:52.541187 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:52.542294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:52.542267 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:52.543622 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:52.543601 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-dkxqn" Nov 04 01:23:55.058903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:55.058868 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h"] Nov 04 01:23:55.109152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:55.109115 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c50093-j2p5h"] Nov 04 01:23:55.351813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:23:55.351783 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4df72c4-7ee5-4b09-abcf-1ecd87359b06" path="/var/lib/kubelet/pods/e4df72c4-7ee5-4b09-abcf-1ecd87359b06/volumes" Nov 04 01:24:09.053633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:09.053583 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz"] Nov 04 01:24:09.096810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:09.096768 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f57b4-v2bcz"] Nov 04 01:24:09.355515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:09.355472 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f7049a-bca5-4750-836d-48b5adb431c8" path="/var/lib/kubelet/pods/00f7049a-bca5-4750-836d-48b5adb431c8/volumes" Nov 04 01:24:11.515119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.515086 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76"] Nov 04 01:24:11.518930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.518913 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.561228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.561196 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.561369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.561321 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.561512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.561497 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgcm\" (UniqueName: \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.611509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.611479 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:24:11.616047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.616002 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:24:11.622663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.622637 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76"] Nov 04 01:24:11.664274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.664240 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.665811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.665776 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.666461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.665632 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.666683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.666662 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgcm\" (UniqueName: \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.670174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.670154 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.721278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.721256 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgcm\" (UniqueName: \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\") pod \"maas-api-revoke-sub-0b406df7-rtg76\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:11.833176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:11.833146 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:12.224313 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:24:12.224264 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e723bb5_36fd_41d1_a160_4169b8fbbb7f.slice/crio-0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16 WatchSource:0}: Error finding container 0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16: Status 404 returned error can't find the container with id 0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16 Nov 04 01:24:12.228085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:12.228058 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76"] Nov 04 01:24:12.796455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:12.796415 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" event={"ID":"1e723bb5-36fd-41d1-a160-4169b8fbbb7f","Type":"ContainerStarted","Data":"d3a9b4962d0283cb69aa2179b22b946e6cb3d62ff9531e0c829748a7487959d5"} Nov 04 01:24:12.796937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:12.796473 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" event={"ID":"1e723bb5-36fd-41d1-a160-4169b8fbbb7f","Type":"ContainerStarted","Data":"0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16"} Nov 04 01:24:12.917088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:12.916966 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" podStartSLOduration=1.916953208 podStartE2EDuration="1.916953208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:12.912572107 +0000 UTC m=+1621.671234269" watchObservedRunningTime="2026-11-04 01:24:12.916953208 +0000 UTC m=+1621.675615369" Nov 04 01:24:13.806862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:13.806824 4486 generic.go:356] "Generic (PLEG): container finished" podID="1e723bb5-36fd-41d1-a160-4169b8fbbb7f" containerID="d3a9b4962d0283cb69aa2179b22b946e6cb3d62ff9531e0c829748a7487959d5" exitCode=0 Nov 04 01:24:13.807360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:13.806904 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" event={"ID":"1e723bb5-36fd-41d1-a160-4169b8fbbb7f","Type":"ContainerDied","Data":"d3a9b4962d0283cb69aa2179b22b946e6cb3d62ff9531e0c829748a7487959d5"} Nov 04 01:24:13.836563 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:24:13.836525 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:24:15.107095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.107019 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:15.196005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.195669 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\") pod \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " Nov 04 01:24:15.196005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.195906 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\") pod \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " Nov 04 01:24:15.196005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.195948 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\" (UniqueName: \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\") pod \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\" (UID: \"1e723bb5-36fd-41d1-a160-4169b8fbbb7f\") " Nov 04 01:24:15.197696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.197651 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca" pod "1e723bb5-36fd-41d1-a160-4169b8fbbb7f" (UID: "1e723bb5-36fd-41d1-a160-4169b8fbbb7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:15.200923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.200899 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca" pod "1e723bb5-36fd-41d1-a160-4169b8fbbb7f" (UID: "1e723bb5-36fd-41d1-a160-4169b8fbbb7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:15.207147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.207121 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm" pod "1e723bb5-36fd-41d1-a160-4169b8fbbb7f" (UID: "1e723bb5-36fd-41d1-a160-4169b8fbbb7f"). InnerVolumeSpecName "kube-api-access-4cgcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:15.298294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.298258 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:15.298799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.298774 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:15.298952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.298939 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cgcm\" (UniqueName: \"kubernetes.io/projected/1e723bb5-36fd-41d1-a160-4169b8fbbb7f-kube-api-access-4cgcm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:15.828608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.828578 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" Nov 04 01:24:15.828608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.828590 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76" event={"ID":"1e723bb5-36fd-41d1-a160-4169b8fbbb7f","Type":"ContainerDied","Data":"0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16"} Nov 04 01:24:15.828819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:15.828645 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1f0f1155e88fc02d0b72b7c35dca6876cbe10fbe6ff1a7dc13aae1029b6d16" Nov 04 01:24:16.291017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.290202 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4"] Nov 04 01:24:16.298256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.293859 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1e723bb5-36fd-41d1-a160-4169b8fbbb7f" containerName="revoke-keys" Nov 04 01:24:16.298256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.293924 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e723bb5-36fd-41d1-a160-4169b8fbbb7f" containerName="revoke-keys" Nov 04 01:24:16.298256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.298225 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e723bb5-36fd-41d1-a160-4169b8fbbb7f" containerName="revoke-keys" Nov 04 01:24:16.303313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.303285 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.432010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.422155 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:24:16.432010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.422288 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:24:16.432010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.423350 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.432010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.423489 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.432010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.423552 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j8f\" (UniqueName: \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.579608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.579565 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.609906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.590710 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4"] Nov 04 01:24:16.609906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.592767 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.609906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.593017 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.609906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.593093 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j8f\" (UniqueName: \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.609906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.609456 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.650061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.649964 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j8f\" (UniqueName: \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\") pod \"maas-api-revoke-keys-e2e-derive-60398d45-42km4\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:16.943341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:16.943246 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:17.464783 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:24:17.464736 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515a538a_dca5_4170_bd86_c00fd1ed1c80.slice/crio-5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f WatchSource:0}: Error finding container 5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f: Status 404 returned error can't find the container with id 5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f Nov 04 01:24:17.471399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:17.471366 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4"] Nov 04 01:24:17.968559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:17.968212 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" event={"ID":"515a538a-dca5-4170-bd86-c00fd1ed1c80","Type":"ContainerStarted","Data":"c779f0a4a09e10652bd11a18498f0917e40c54df5da119ad0d4aacb78e7cc910"} Nov 04 01:24:17.968559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:17.968278 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" event={"ID":"515a538a-dca5-4170-bd86-c00fd1ed1c80","Type":"ContainerStarted","Data":"5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f"} Nov 04 01:24:18.098956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:18.098902 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" podStartSLOduration=2.098886121 podStartE2EDuration="2.098886121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:18.094931209 +0000 UTC m=+1626.853593370" watchObservedRunningTime="2026-11-04 01:24:18.098886121 +0000 UTC m=+1626.857548316" Nov 04 01:24:18.979208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:18.979118 4486 generic.go:356] "Generic (PLEG): container finished" podID="515a538a-dca5-4170-bd86-c00fd1ed1c80" containerID="c779f0a4a09e10652bd11a18498f0917e40c54df5da119ad0d4aacb78e7cc910" exitCode=0 Nov 04 01:24:18.979635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:18.979211 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" event={"ID":"515a538a-dca5-4170-bd86-c00fd1ed1c80","Type":"ContainerDied","Data":"c779f0a4a09e10652bd11a18498f0917e40c54df5da119ad0d4aacb78e7cc910"} Nov 04 01:24:20.221209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.221181 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:20.407906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.407866 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\") pod \"515a538a-dca5-4170-bd86-c00fd1ed1c80\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " Nov 04 01:24:20.408119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.407996 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\") pod \"515a538a-dca5-4170-bd86-c00fd1ed1c80\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " Nov 04 01:24:20.408226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.408204 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\" (UniqueName: \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\") pod \"515a538a-dca5-4170-bd86-c00fd1ed1c80\" (UID: \"515a538a-dca5-4170-bd86-c00fd1ed1c80\") " Nov 04 01:24:20.408646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.408620 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca" pod "515a538a-dca5-4170-bd86-c00fd1ed1c80" (UID: "515a538a-dca5-4170-bd86-c00fd1ed1c80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:20.409053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.409024 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:20.411005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.410970 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca" pod "515a538a-dca5-4170-bd86-c00fd1ed1c80" (UID: "515a538a-dca5-4170-bd86-c00fd1ed1c80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:20.413324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.413305 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f" pod "515a538a-dca5-4170-bd86-c00fd1ed1c80" (UID: "515a538a-dca5-4170-bd86-c00fd1ed1c80"). InnerVolumeSpecName "kube-api-access-w2j8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:20.511755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.511186 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515a538a-dca5-4170-bd86-c00fd1ed1c80-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:20.511755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.511216 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j8f\" (UniqueName: \"kubernetes.io/projected/515a538a-dca5-4170-bd86-c00fd1ed1c80-kube-api-access-w2j8f\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:20.526345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.526317 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc"] Nov 04 01:24:20.527136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.527117 4486 cpu_manager.go:414] "containerMap: removing container" podUID="515a538a-dca5-4170-bd86-c00fd1ed1c80" containerName="revoke-keys" Nov 04 01:24:20.527197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.527136 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515a538a-dca5-4170-bd86-c00fd1ed1c80" containerName="revoke-keys" Nov 04 01:24:20.527276 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.527267 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="515a538a-dca5-4170-bd86-c00fd1ed1c80" containerName="revoke-keys" Nov 04 01:24:20.564090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.560253 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.622220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.622188 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc"] Nov 04 01:24:20.717091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.716976 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5v9w\" (UniqueName: \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.717091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.717045 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.717337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.717272 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.819621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.818721 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5v9w\" (UniqueName: \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.819621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.818783 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.819621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.819060 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.819621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.819531 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.823406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.823380 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.878070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.878037 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5v9w\" (UniqueName: \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\") pod \"maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:20.886050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:20.886027 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:21.009231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:21.009195 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" event={"ID":"515a538a-dca5-4170-bd86-c00fd1ed1c80","Type":"ContainerDied","Data":"5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f"} Nov 04 01:24:21.009385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:21.009259 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7fed0034021bbf90ac8819dade3cec3f252055c7e5c712aa7efe79a77f773f" Nov 04 01:24:21.009385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:21.009326 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4" Nov 04 01:24:21.308256 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:24:21.308222 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9413b371_c8ae_40da_a3f9_7762387e83fd.slice/crio-2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96 WatchSource:0}: Error finding container 2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96: Status 404 returned error can't find the container with id 2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96 Nov 04 01:24:21.312108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:21.311871 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc"] Nov 04 01:24:22.022587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:22.022470 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" event={"ID":"9413b371-c8ae-40da-a3f9-7762387e83fd","Type":"ContainerStarted","Data":"8b0a318cc4ba393840496bda6423b061b1e5cae8adcc82c6c0cf370479c36bc9"} Nov 04 01:24:22.022587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:22.022540 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" event={"ID":"9413b371-c8ae-40da-a3f9-7762387e83fd","Type":"ContainerStarted","Data":"2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96"} Nov 04 01:24:22.150065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:22.146022 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" podStartSLOduration=2.145977537 podStartE2EDuration="2.145977537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:22.145391507 +0000 UTC m=+1630.904053670" watchObservedRunningTime="2026-11-04 01:24:22.145977537 +0000 UTC m=+1630.904639699" Nov 04 01:24:23.036055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:23.036002 4486 generic.go:356] "Generic (PLEG): container finished" podID="9413b371-c8ae-40da-a3f9-7762387e83fd" containerID="8b0a318cc4ba393840496bda6423b061b1e5cae8adcc82c6c0cf370479c36bc9" exitCode=0 Nov 04 01:24:23.036526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:23.036191 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" event={"ID":"9413b371-c8ae-40da-a3f9-7762387e83fd","Type":"ContainerDied","Data":"8b0a318cc4ba393840496bda6423b061b1e5cae8adcc82c6c0cf370479c36bc9"} Nov 04 01:24:24.295896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.295865 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:24.333202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.333169 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\") pod \"9413b371-c8ae-40da-a3f9-7762387e83fd\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " Nov 04 01:24:24.333444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.333421 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\" (UniqueName: \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\") pod \"9413b371-c8ae-40da-a3f9-7762387e83fd\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " Nov 04 01:24:24.333533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.333516 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\") pod \"9413b371-c8ae-40da-a3f9-7762387e83fd\" (UID: \"9413b371-c8ae-40da-a3f9-7762387e83fd\") " Nov 04 01:24:24.334510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.334378 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca" pod "9413b371-c8ae-40da-a3f9-7762387e83fd" (UID: "9413b371-c8ae-40da-a3f9-7762387e83fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:24.334666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.334630 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:24.335965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.335926 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca" pod "9413b371-c8ae-40da-a3f9-7762387e83fd" (UID: "9413b371-c8ae-40da-a3f9-7762387e83fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:24.336275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.336250 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w" pod "9413b371-c8ae-40da-a3f9-7762387e83fd" (UID: "9413b371-c8ae-40da-a3f9-7762387e83fd"). InnerVolumeSpecName "kube-api-access-x5v9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:24.438744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.438706 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5v9w\" (UniqueName: \"kubernetes.io/projected/9413b371-c8ae-40da-a3f9-7762387e83fd-kube-api-access-x5v9w\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:24.438744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.438744 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9413b371-c8ae-40da-a3f9-7762387e83fd-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:24.851878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.851837 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb"] Nov 04 01:24:24.855105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.855077 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9413b371-c8ae-40da-a3f9-7762387e83fd" containerName="revoke-keys" Nov 04 01:24:24.855105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.855105 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9413b371-c8ae-40da-a3f9-7762387e83fd" containerName="revoke-keys" Nov 04 01:24:24.855331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.855317 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9413b371-c8ae-40da-a3f9-7762387e83fd" containerName="revoke-keys" Nov 04 01:24:24.960873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.960747 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb"] Nov 04 01:24:24.961581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:24.961134 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.058171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.058108 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.058450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.058421 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqchz\" (UniqueName: \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.058657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.058630 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.081164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.081129 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" Nov 04 01:24:25.081164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.081152 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc" event={"ID":"9413b371-c8ae-40da-a3f9-7762387e83fd","Type":"ContainerDied","Data":"2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96"} Nov 04 01:24:25.081379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.081207 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1ae69811a00d21d4c1fabde8f2924e9c0f5ad5dd2b081a84e3ed9c1bc0cd96" Nov 04 01:24:25.160671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.160592 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.160840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.160792 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqchz\" (UniqueName: \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.160926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.160910 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.161446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.161404 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.163652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.163630 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.218302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.218272 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqchz\" (UniqueName: \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\") pod \"maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.381905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.381870 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:25.872238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:25.872205 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb"] Nov 04 01:24:25.875906 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:24:25.875053 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde826c83_baa3_492b_b4ea_3a4fa971ae40.slice/crio-e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0 WatchSource:0}: Error finding container e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0: Status 404 returned error can't find the container with id e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0 Nov 04 01:24:26.104303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:26.104253 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" event={"ID":"de826c83-baa3-492b-b4ea-3a4fa971ae40","Type":"ContainerStarted","Data":"e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0"} Nov 04 01:24:27.115697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:27.115652 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" event={"ID":"de826c83-baa3-492b-b4ea-3a4fa971ae40","Type":"ContainerStarted","Data":"30b311e5a4a1b87c4c8554843dbb40b8c62d8078016cdf18ad3cc70e56529d16"} Nov 04 01:24:27.228939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:27.228869 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" podStartSLOduration=3.228849838 podStartE2EDuration="3.228849838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:27.225559797 +0000 UTC m=+1635.984221997" watchObservedRunningTime="2026-11-04 01:24:27.228849838 +0000 UTC m=+1635.987512000" Nov 04 01:24:28.129577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.129540 4486 generic.go:356] "Generic (PLEG): container finished" podID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerID="3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff" exitCode=0 Nov 04 01:24:28.130143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.129620 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerDied","Data":"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff"} Nov 04 01:24:28.130233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.130198 4486 scope.go:122] "RemoveContainer" containerID="3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff" Nov 04 01:24:28.131727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.131702 4486 generic.go:356] "Generic (PLEG): container finished" podID="de826c83-baa3-492b-b4ea-3a4fa971ae40" containerID="30b311e5a4a1b87c4c8554843dbb40b8c62d8078016cdf18ad3cc70e56529d16" exitCode=0 Nov 04 01:24:28.131835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.131785 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" event={"ID":"de826c83-baa3-492b-b4ea-3a4fa971ae40","Type":"ContainerDied","Data":"30b311e5a4a1b87c4c8554843dbb40b8c62d8078016cdf18ad3cc70e56529d16"} Nov 04 01:24:28.132109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.132081 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:24:28.132185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.132169 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:24:28.132227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:28.132217 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:24:29.164440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.164394 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerStarted","Data":"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682"} Nov 04 01:24:29.547065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.544165 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:29.720191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.708938 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\") pod \"de826c83-baa3-492b-b4ea-3a4fa971ae40\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " Nov 04 01:24:29.720191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.709094 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\") pod \"de826c83-baa3-492b-b4ea-3a4fa971ae40\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " Nov 04 01:24:29.720191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.709208 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\" (UniqueName: \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\") pod \"de826c83-baa3-492b-b4ea-3a4fa971ae40\" (UID: \"de826c83-baa3-492b-b4ea-3a4fa971ae40\") " Nov 04 01:24:29.720191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.711046 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca" pod "de826c83-baa3-492b-b4ea-3a4fa971ae40" (UID: "de826c83-baa3-492b-b4ea-3a4fa971ae40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:29.726652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.724066 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz" pod "de826c83-baa3-492b-b4ea-3a4fa971ae40" (UID: "de826c83-baa3-492b-b4ea-3a4fa971ae40"). InnerVolumeSpecName "kube-api-access-sqchz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:29.824043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.823574 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca" pod "de826c83-baa3-492b-b4ea-3a4fa971ae40" (UID: "de826c83-baa3-492b-b4ea-3a4fa971ae40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862330 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862372 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqchz\" (UniqueName: \"kubernetes.io/projected/de826c83-baa3-492b-b4ea-3a4fa971ae40-kube-api-access-sqchz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862388 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de826c83-baa3-492b-b4ea-3a4fa971ae40-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862420 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862541 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:29.898904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:29.862611 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:30.175639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:30.175601 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" Nov 04 01:24:30.175639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:30.175629 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb" event={"ID":"de826c83-baa3-492b-b4ea-3a4fa971ae40","Type":"ContainerDied","Data":"e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0"} Nov 04 01:24:30.176284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:30.175682 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d28a50e49e429bb9235869f49a673cbe4b062916c06320a52c6262223d53c0" Nov 04 01:24:30.768902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:30.768398 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:30.768902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:30.768464 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:31.763664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:31.763632 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:31.764118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:31.763687 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:32.794008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:32.778677 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:32.794008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:32.778735 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:33.763478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:33.763438 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:33.763667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:33.763495 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:34.531010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:34.530138 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:24:34.763648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:34.763604 4486 patch_prober.go:29] interesting pod/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Nov 04 01:24:34.763823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:34.763674 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Nov 04 01:24:35.768963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:35.768933 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:24:35.769948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:35.769931 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:24:36.052844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:36.052762 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8"] Nov 04 01:24:36.058743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:36.058721 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a738a44-9trt8"] Nov 04 01:24:37.350643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:37.350606 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cd435d-8b8c-4513-a1bd-c6c96120e4d9" path="/var/lib/kubelet/pods/a9cd435d-8b8c-4513-a1bd-c6c96120e4d9/volumes" Nov 04 01:24:43.832925 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:24:43.832875 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:24:53.127474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:53.127251 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd"] Nov 04 01:24:53.194926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:53.194890 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fd8de4-mccwd"] Nov 04 01:24:53.426085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:24:53.420070 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe" path="/var/lib/kubelet/pods/c4acfb06-4ed1-4b6b-b8bc-cf9b71cdbebe/volumes" Nov 04 01:25:04.801104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:04.801064 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4"] Nov 04 01:25:04.809903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:04.809855 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-60398d45-42km4"] Nov 04 01:25:05.366123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:05.366081 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515a538a-dca5-4170-bd86-c00fd1ed1c80" path="/var/lib/kubelet/pods/515a538a-dca5-4170-bd86-c00fd1ed1c80/volumes" Nov 04 01:25:06.275707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.275657 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc"] Nov 04 01:25:06.315430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.315264 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e400a3a6-gl5vc"] Nov 04 01:25:06.326576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.326537 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:25:06.329102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.326769 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" containerID="cri-o://64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37" gracePeriod=30 Nov 04 01:25:06.512184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.512147 4486 patch_prober.go:29] interesting pod/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:25:06.512589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:06.512480 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:25:07.352483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:07.352446 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9413b371-c8ae-40da-a3f9-7762387e83fd" path="/var/lib/kubelet/pods/9413b371-c8ae-40da-a3f9-7762387e83fd/volumes" Nov 04 01:25:07.844605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:07.821109 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb"] Nov 04 01:25:07.844605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:07.835883 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b45d1fe0-bj4sb"] Nov 04 01:25:07.868841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:07.868790 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:25:07.869811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:07.869086 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" containerID="cri-o://f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682" gracePeriod=30 Nov 04 01:25:09.351119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:09.351083 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de826c83-baa3-492b-b4ea-3a4fa971ae40" path="/var/lib/kubelet/pods/de826c83-baa3-492b-b4ea-3a4fa971ae40/volumes" Nov 04 01:25:11.629812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.629746 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:25:11.733663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.733629 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.733853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.733737 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.733853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.733829 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.733978 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734126 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734152 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:11.734217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734173 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734161 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:11.734375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734221 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734331 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734362 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\") pod \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\" (UID: \"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34\") " Nov 04 01:25:11.734523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734504 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:11.734575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734520 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:11.734708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.734667 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:11.735762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.735149 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.735955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.735817 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.735955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.735930 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.736351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.736156 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.736835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.736310 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.736835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.736688 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:11.736835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.736768 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "kube-api-access-qtzhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:11.737684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.737658 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:11.737938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.737910 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo" pod "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" (UID: "d4d46b0d-aad2-47af-8fba-d73cd4ab1c34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:11.838739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.838335 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzhh\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-kube-api-access-qtzhh\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.838739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.838372 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.838739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.838390 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.838739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.838405 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:11.926081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.925947 4486 generic.go:356] "Generic (PLEG): container finished" podID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerID="64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37" exitCode=0 Nov 04 01:25:11.926081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.926048 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" Nov 04 01:25:11.927095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.927056 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerDied","Data":"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37"} Nov 04 01:25:11.927224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.927138 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7" event={"ID":"d4d46b0d-aad2-47af-8fba-d73cd4ab1c34","Type":"ContainerDied","Data":"c5406801f6aa6486e19119c9d76ebd7b7df40a1cc8ffc45a9adf176a8280719d"} Nov 04 01:25:11.927224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.927173 4486 scope.go:122] "RemoveContainer" containerID="64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37" Nov 04 01:25:11.941370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.941347 4486 scope.go:122] "RemoveContainer" containerID="215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2" Nov 04 01:25:11.954613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.954590 4486 scope.go:122] "RemoveContainer" containerID="64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37" Nov 04 01:25:11.954932 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:11.954911 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37\": container with ID starting with 64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37 not found: ID does not exist" containerID="64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37" Nov 04 01:25:11.955021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.954945 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37"} err="failed to get container status \"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37\": rpc error: code = NotFound desc = could not find container \"64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37\": container with ID starting with 64d50362232531ad739fae3496e16d6195aa4e76112b6b8d564dbfcb11db8a37 not found: ID does not exist" Nov 04 01:25:11.955021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.954970 4486 scope.go:122] "RemoveContainer" containerID="215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2" Nov 04 01:25:11.955269 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:11.955249 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2\": container with ID starting with 215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2 not found: ID does not exist" containerID="215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2" Nov 04 01:25:11.955330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:11.955273 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2"} err="failed to get container status \"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2\": rpc error: code = NotFound desc = could not find container \"215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2\": container with ID starting with 215d5b5196db362f49a4cb628e053c814fa70820feae92ec482194e5bcc32bd2 not found: ID does not exist" Nov 04 01:25:12.088859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:12.088822 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:25:12.094923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:12.094888 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e400a3a6-openshift-default-d9fc7fc9c-tn7j7"] Nov 04 01:25:13.191173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.191147 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:25:13.353365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.353322 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" path="/var/lib/kubelet/pods/d4d46b0d-aad2-47af-8fba-d73cd4ab1c34/volumes" Nov 04 01:25:13.363168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363131 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363197 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363231 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363277 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363341 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363400 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363421 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363518 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363541 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\") pod \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\" (UID: \"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe\") " Nov 04 01:25:13.363794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363595 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:13.363794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363619 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:13.363794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363648 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:13.363794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363654 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:13.364012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.363878 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:13.364356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.364325 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.364356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.364346 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.364356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.364356 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.364581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.364371 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.364581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.364381 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.365792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.365760 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:13.365906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.365837 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "kube-api-access-p64cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:13.366150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.366128 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:13.366210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.366150 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token" pod "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" (UID: "4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:13.465906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.465829 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.465906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.465860 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.465906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.465870 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64cv\" (UniqueName: \"kubernetes.io/projected/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-kube-api-access-p64cv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.465906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.465881 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:13.875219 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:13.875172 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:25:13.950780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.950742 4486 generic.go:356] "Generic (PLEG): container finished" podID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerID="f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682" exitCode=0 Nov 04 01:25:13.950959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.950793 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" Nov 04 01:25:13.950959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.950811 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerDied","Data":"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682"} Nov 04 01:25:13.950959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.950885 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h" event={"ID":"4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe","Type":"ContainerDied","Data":"e43613d4a9f3413331dce6d04aff793dbf8321204b218d59dbb1f0d12bc900f6"} Nov 04 01:25:13.950959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.950917 4486 scope.go:122] "RemoveContainer" containerID="f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682" Nov 04 01:25:13.965033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.964971 4486 scope.go:122] "RemoveContainer" containerID="3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff" Nov 04 01:25:13.975832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.975812 4486 scope.go:122] "RemoveContainer" containerID="f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682" Nov 04 01:25:13.976160 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:13.976136 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682\": container with ID starting with f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682 not found: ID does not exist" containerID="f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682" Nov 04 01:25:13.976236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.976168 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682"} err="failed to get container status \"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682\": rpc error: code = NotFound desc = could not find container \"f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682\": container with ID starting with f00aecf8804775fe0068836e7cd77ef674e6e70ced6967aa56fdd6ebb95b2682 not found: ID does not exist" Nov 04 01:25:13.976236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.976185 4486 scope.go:122] "RemoveContainer" containerID="3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff" Nov 04 01:25:13.976433 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:13.976415 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff\": container with ID starting with 3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff not found: ID does not exist" containerID="3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff" Nov 04 01:25:13.976490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:13.976440 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff"} err="failed to get container status \"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff\": rpc error: code = NotFound desc = could not find container \"3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff\": container with ID starting with 3791679e7506b72786d92f44af15b5e112d3de4a07c9cd2027722c7099b8d4ff not found: ID does not exist" Nov 04 01:25:14.119835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:14.119801 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:25:14.167184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:14.167107 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b45d1fe0-gw-openshift-default-9bb586765-x6l7h"] Nov 04 01:25:15.352371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:15.352333 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" path="/var/lib/kubelet/pods/4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe/volumes" Nov 04 01:25:18.517764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:18.517722 4486 scope.go:122] "RemoveContainer" containerID="37ff6433b1f99b1308ad478e5b077440220678d470d73d3ea5b0d58a997ce73c" Nov 04 01:25:18.543917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:18.542961 4486 scope.go:122] "RemoveContainer" containerID="d274a434420e3a124708a7b9b12cbdf6aa2f848e0a33956e0eb52aa05bc0ff5a" Nov 04 01:25:19.049478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:19.049439 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77"] Nov 04 01:25:19.095976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:19.095941 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb0c968-zxh77"] Nov 04 01:25:19.353835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:19.353794 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf9b378-6f7e-4728-9c7b-8510d706898d" path="/var/lib/kubelet/pods/3bf9b378-6f7e-4728-9c7b-8510d706898d/volumes" Nov 04 01:25:24.054156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:24.054119 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq"] Nov 04 01:25:24.097839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:24.097809 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c615a2-9vwjq"] Nov 04 01:25:25.353295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:25.353256 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71ddd67-1b6b-440c-bdeb-e2cc9f088459" path="/var/lib/kubelet/pods/c71ddd67-1b6b-440c-bdeb-e2cc9f088459/volumes" Nov 04 01:25:39.049613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:39.049564 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz"] Nov 04 01:25:39.096688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:39.096647 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e17c15cb-5rnxz"] Nov 04 01:25:39.351303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:39.351267 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ea1ab2-3743-406b-8c44-aa929f147f1e" path="/var/lib/kubelet/pods/41ea1ab2-3743-406b-8c44-aa929f147f1e/volumes" Nov 04 01:25:43.833277 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:25:43.833228 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:25:51.091943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.091903 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb"] Nov 04 01:25:51.098553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.098517 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d83b6d4e-7cshb"] Nov 04 01:25:51.423020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.422909 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37aaabab-a61e-43a2-a92f-6b5a2517b1ab" path="/var/lib/kubelet/pods/37aaabab-a61e-43a2-a92f-6b5a2517b1ab/volumes" Nov 04 01:25:51.505236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.505199 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:25:51.506023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506004 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506022 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506035 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506041 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506053 4486 cpu_manager.go:414] "containerMap: removing container" podUID="de826c83-baa3-492b-b4ea-3a4fa971ae40" containerName="revoke-keys" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506058 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de826c83-baa3-492b-b4ea-3a4fa971ae40" containerName="revoke-keys" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506087 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506092 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506217 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506227 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:51.506519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506235 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcca5ce-72c2-4b4e-94db-76ea99d9f0fe" containerName="istio-proxy" Nov 04 01:25:51.506519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.506242 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="de826c83-baa3-492b-b4ea-3a4fa971ae40" containerName="revoke-keys" Nov 04 01:25:51.511222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.511194 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.561372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.561330 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9a0432-openshift-default-dockercfg-765cl\"" Nov 04 01:25:51.604504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.604473 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641549 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641687 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641730 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmcw\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641775 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641851 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.641910 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.642038 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.642065 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.642601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.642126 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.744663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.744329 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.745051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.745025 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.745291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.745270 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmcw\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.745765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.745694 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.745876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.745845 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.745939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.745916 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.746142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.746118 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.746340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.746310 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.746495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.746480 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.748132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.747846 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.748561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.748541 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.748864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.748844 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.764187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.764153 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.764483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.764344 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.770702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.767271 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.787564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.786784 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.804454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.804420 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.805110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.805080 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmcw\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\") pod \"e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:51.830506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:51.830455 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:52.284960 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:25:52.284928 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcefdeb_c027_4a22_a40f_20f2a5c765c8.slice/crio-43d5fa80134559c00785713e8291266a3d054470eeff355e0a60f91078c1ac50 WatchSource:0}: Error finding container 43d5fa80134559c00785713e8291266a3d054470eeff355e0a60f91078c1ac50: Status 404 returned error can't find the container with id 43d5fa80134559c00785713e8291266a3d054470eeff355e0a60f91078c1ac50 Nov 04 01:25:52.287418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.287392 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:25:52.287687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.287659 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:52.287724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.287716 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:52.287758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.287743 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:52.456149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.456105 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerStarted","Data":"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285"} Nov 04 01:25:52.456337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.456174 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerStarted","Data":"43d5fa80134559c00785713e8291266a3d054470eeff355e0a60f91078c1ac50"} Nov 04 01:25:52.590397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.590334 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podStartSLOduration=1.590316494 podStartE2EDuration="1.590316494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:52.583876761 +0000 UTC m=+1721.342538904" watchObservedRunningTime="2026-11-04 01:25:52.590316494 +0000 UTC m=+1721.348978656" Nov 04 01:25:52.830696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:52.830662 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:53.836140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:53.836104 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:54.532141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.532110 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:54.533047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.533024 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:25:54.551417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.551387 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:25:54.552165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.552144 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:54.552165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.552165 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:54.552346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.552312 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d46b0d-aad2-47af-8fba-d73cd4ab1c34" containerName="istio-proxy" Nov 04 01:25:54.556751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.556728 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.606717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.606675 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8aa9f3a2-openshift-default-dockercfg-cwtwm\"" Nov 04 01:25:54.658968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.657171 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:25:54.714162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714124 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714272 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714360 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gsgw\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714451 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714482 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714638 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714704 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714731 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.714818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.714798 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.816775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.816740 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.816793 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.816945 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817010 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817032 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817088 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817188 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817225 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817267 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817280 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gsgw\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817670 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.817896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.817877 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.818525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.818493 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.818879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.818855 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.820123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.820098 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.820277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.820258 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.873531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.873488 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:54.874035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:54.873610 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gsgw\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\") pod \"e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:55.176336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:55.176244 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:55.615660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:55.615623 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:25:55.646365 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:25:55.646335 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92bf487_dd47_4d52_8aa9_9258f2ba92d0.slice/crio-b846d22a63d2a6faded116b123ad557502e8a19472217745d4affb129e231a13 WatchSource:0}: Error finding container b846d22a63d2a6faded116b123ad557502e8a19472217745d4affb129e231a13: Status 404 returned error can't find the container with id b846d22a63d2a6faded116b123ad557502e8a19472217745d4affb129e231a13 Nov 04 01:25:55.651444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:55.651407 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:55.651538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:55.651482 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:55.651538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:55.651519 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:25:56.557862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:56.557823 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerStarted","Data":"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95"} Nov 04 01:25:56.558483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:56.557891 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerStarted","Data":"b846d22a63d2a6faded116b123ad557502e8a19472217745d4affb129e231a13"} Nov 04 01:25:56.724789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:56.724730 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podStartSLOduration=2.7247163309999998 podStartE2EDuration="2.724716331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:56.672200812 +0000 UTC m=+1725.430862976" watchObservedRunningTime="2026-11-04 01:25:56.724716331 +0000 UTC m=+1725.483378567" Nov 04 01:25:57.177378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:57.177341 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:57.182810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:57.182783 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:57.570636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:57.570602 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:25:57.571756 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:25:57.571734 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:26:04.056186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:04.056153 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv"] Nov 04 01:26:04.104787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:04.104747 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd39b8ef-5fzsv"] Nov 04 01:26:05.050958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.050924 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc"] Nov 04 01:26:05.056457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.056431 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20839ace-xvqjc"] Nov 04 01:26:05.351716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.351675 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a79bb77-1b36-4917-bcc9-90b877a937a3" path="/var/lib/kubelet/pods/0a79bb77-1b36-4917-bcc9-90b877a937a3/volumes" Nov 04 01:26:05.352334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.352311 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee56a7d-4a74-47c0-bba9-d4c3618ac499" path="/var/lib/kubelet/pods/fee56a7d-4a74-47c0-bba9-d4c3618ac499/volumes" Nov 04 01:26:05.524628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.524594 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:26:05.528488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.528470 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.563172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.563138 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.563760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.563690 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.563895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.563805 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.563895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.563845 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.564074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.563924 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.564134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.564118 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.564201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.564187 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.564284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.564267 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.564352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.564341 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjn2\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.651060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.650280 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5186fe-openshift-default-dockercfg-9lcnx\"" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684123 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjn2\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684214 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684313 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684367 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684401 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684479 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684643 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684710 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.684785 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.699416 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.700413 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.704051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.700655 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.735006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.705468 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.735006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.705818 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.867959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.867891 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.893699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.893629 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.902592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.898128 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:26:05.932214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.932089 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.947007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.946754 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjn2\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\") pod \"e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:05.993365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:05.993330 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:06.459669 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:26:06.459637 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5d3b9d_d8ea_4c19_a872_cd9b656fde6c.slice/crio-82d356411a6cab0907dc022a1a9f57bc9a5c0cb52be7375428931072fa424f76 WatchSource:0}: Error finding container 82d356411a6cab0907dc022a1a9f57bc9a5c0cb52be7375428931072fa424f76: Status 404 returned error can't find the container with id 82d356411a6cab0907dc022a1a9f57bc9a5c0cb52be7375428931072fa424f76 Nov 04 01:26:06.462646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.462609 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:26:06.462759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.462681 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:26:06.462759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.462709 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:26:06.468714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.463606 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:26:06.834761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.834718 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerStarted","Data":"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6"} Nov 04 01:26:06.834957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.834788 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerStarted","Data":"82d356411a6cab0907dc022a1a9f57bc9a5c0cb52be7375428931072fa424f76"} Nov 04 01:26:06.956231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.956167 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podStartSLOduration=1.95614558 podStartE2EDuration="1.95614558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:06.948462553 +0000 UTC m=+1735.707124715" watchObservedRunningTime="2026-11-04 01:26:06.95614558 +0000 UTC m=+1735.714807742" Nov 04 01:26:06.994998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:06.994957 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:08.000763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:08.000730 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:08.855129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:08.855095 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:08.855957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:08.855939 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:26:10.053938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:10.053897 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k"] Nov 04 01:26:10.106928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:10.106887 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d3c9da-t8w4k"] Nov 04 01:26:11.420636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:11.420582 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa613ab-3dc3-404e-bacb-1c07a9fe8fda" path="/var/lib/kubelet/pods/2fa613ab-3dc3-404e-bacb-1c07a9fe8fda/volumes" Nov 04 01:26:13.856698 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:26:13.856652 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:26:14.173157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:14.173020 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42"] Nov 04 01:26:14.185762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:14.184923 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddfcbf04-kkv42"] Nov 04 01:26:15.355251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:15.355216 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d9b722-2852-4c06-89c4-ee13c845d2a0" path="/var/lib/kubelet/pods/f2d9b722-2852-4c06-89c4-ee13c845d2a0/volumes" Nov 04 01:26:16.057533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:16.057496 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4"] Nov 04 01:26:16.104466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:16.104425 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7"] Nov 04 01:26:16.110093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:16.110064 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd61dd02-p87q7"] Nov 04 01:26:16.114769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:16.114748 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40faa50-hkvw4"] Nov 04 01:26:17.353347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:17.353301 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ab2709-b5c2-43c7-a362-8b826e670f9f" path="/var/lib/kubelet/pods/00ab2709-b5c2-43c7-a362-8b826e670f9f/volumes" Nov 04 01:26:17.353995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:17.353959 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3fadf6-120b-412a-9127-e104075957f1" path="/var/lib/kubelet/pods/2a3fadf6-120b-412a-9127-e104075957f1/volumes" Nov 04 01:26:18.624500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:18.624465 4486 scope.go:122] "RemoveContainer" containerID="7e1ac79498d1df70f9cb2d21f352237b0db73108e02720dae3ae27abd6b7609d" Nov 04 01:26:18.637904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:18.637875 4486 scope.go:122] "RemoveContainer" containerID="8506bd5427be5e7c9b839c6baf295c0ec9bc5ca61f3cc7114550d62904fddaee" Nov 04 01:26:18.652551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:18.652510 4486 scope.go:122] "RemoveContainer" containerID="17d2b3d5f6545aebb47be95848802390a0ea4f94c0b5cefbe2da6c3f1a55c353" Nov 04 01:26:24.090290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:24.090199 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn"] Nov 04 01:26:24.134773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:24.134736 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5feeb212-hq2jn"] Nov 04 01:26:25.351532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:25.351489 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe567b29-b506-4583-9587-b4e9f7fa21b7" path="/var/lib/kubelet/pods/fe567b29-b506-4583-9587-b4e9f7fa21b7/volumes" Nov 04 01:26:29.058924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:29.058889 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h"] Nov 04 01:26:29.105858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:29.105828 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5c69b5-6nq6h"] Nov 04 01:26:29.354025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:29.353974 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b148509-0a4e-4173-b1f5-ac46ff4e7fa2" path="/var/lib/kubelet/pods/6b148509-0a4e-4173-b1f5-ac46ff4e7fa2/volumes" Nov 04 01:26:33.068655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:33.068615 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9"] Nov 04 01:26:33.114760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:33.114726 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8b609b5-m25b9"] Nov 04 01:26:33.353351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:33.353272 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72014b47-7137-4ac4-a3f6-8ce3dd8901cd" path="/var/lib/kubelet/pods/72014b47-7137-4ac4-a3f6-8ce3dd8901cd/volumes" Nov 04 01:26:34.066205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:34.066163 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w"] Nov 04 01:26:34.110088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:34.110057 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7faf423c-hr72w"] Nov 04 01:26:34.115639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:34.115611 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc"] Nov 04 01:26:34.120252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:34.120229 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70447975-z8sdc"] Nov 04 01:26:35.057066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:35.057024 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t"] Nov 04 01:26:35.100837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:35.100794 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a909cc-gjt8t"] Nov 04 01:26:35.353247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:35.353213 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec1488c-9c35-4489-8d91-a774b21c1218" path="/var/lib/kubelet/pods/2ec1488c-9c35-4489-8d91-a774b21c1218/volumes" Nov 04 01:26:35.353723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:35.353707 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386a2927-c251-44aa-9952-003291c700c8" path="/var/lib/kubelet/pods/386a2927-c251-44aa-9952-003291c700c8/volumes" Nov 04 01:26:35.354169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:35.354155 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b17593-8487-4789-8923-48be2daada79" path="/var/lib/kubelet/pods/94b17593-8487-4789-8923-48be2daada79/volumes" Nov 04 01:26:36.053602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:36.053564 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c"] Nov 04 01:26:36.097484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:36.097451 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8517a231-6wq7c"] Nov 04 01:26:37.356198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:37.356157 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3151800-d67b-4e59-93fd-ec9eb5e23f74" path="/var/lib/kubelet/pods/d3151800-d67b-4e59-93fd-ec9eb5e23f74/volumes" Nov 04 01:26:43.831353 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:26:43.831312 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:26:44.048126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:44.048091 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg"] Nov 04 01:26:44.094651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:44.094570 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce3bf3c-ljqlg"] Nov 04 01:26:45.049900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.049867 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255"] Nov 04 01:26:45.102048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.102010 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a55a0f-jb255"] Nov 04 01:26:45.111627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.111596 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs"] Nov 04 01:26:45.116720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.116697 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe00dfd5-dtmhs"] Nov 04 01:26:45.351132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.351098 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d324d7-9570-4737-9c6d-f332d2f77a1e" path="/var/lib/kubelet/pods/a0d324d7-9570-4737-9c6d-f332d2f77a1e/volumes" Nov 04 01:26:45.351641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.351618 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb4f115-36b5-4fea-838a-958f88ef72f8" path="/var/lib/kubelet/pods/bcb4f115-36b5-4fea-838a-958f88ef72f8/volumes" Nov 04 01:26:45.352108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:45.352093 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1ee5ac-7627-43b7-b72e-131ab52070e5" path="/var/lib/kubelet/pods/cf1ee5ac-7627-43b7-b72e-131ab52070e5/volumes" Nov 04 01:26:47.063939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:47.063904 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc"] Nov 04 01:26:47.110878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:47.110846 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f110a619-2rlhc"] Nov 04 01:26:47.356437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:47.356353 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb9f6e1-2995-4a2f-9923-aea53664f57f" path="/var/lib/kubelet/pods/7fb9f6e1-2995-4a2f-9923-aea53664f57f/volumes" Nov 04 01:26:48.049470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:48.049435 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn"] Nov 04 01:26:48.096332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:48.096293 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5aab579-9cfsn"] Nov 04 01:26:49.352178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:49.352148 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dbe76d-3225-46bf-8f1b-fef4b74dc31c" path="/var/lib/kubelet/pods/67dbe76d-3225-46bf-8f1b-fef4b74dc31c/volumes" Nov 04 01:26:57.054240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:57.054200 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d"] Nov 04 01:26:57.101628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:57.101595 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e454888a-2np9d"] Nov 04 01:26:57.352154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:57.352115 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6658141e-666b-4e5f-b6a3-ebb73346e0f3" path="/var/lib/kubelet/pods/6658141e-666b-4e5f-b6a3-ebb73346e0f3/volumes" Nov 04 01:26:58.051092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:58.051054 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l"] Nov 04 01:26:58.098479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:58.098430 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac6d1e3-hmw2l"] Nov 04 01:26:59.351864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:26:59.351831 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b565503-8053-4944-9cfc-a9a068ba8cb1" path="/var/lib/kubelet/pods/9b565503-8053-4944-9cfc-a9a068ba8cb1/volumes" Nov 04 01:27:01.982812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:01.982769 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p"] Nov 04 01:27:01.988239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:01.988217 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.042378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.042344 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p"] Nov 04 01:27:02.080965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.080926 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:27:02.081163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.081020 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:27:02.130789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.130754 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99d4\" (UniqueName: \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.130967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.130885 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.130967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.130938 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.232887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.232784 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99d4\" (UniqueName: \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.233088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.232909 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.233088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.233012 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.234117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.234076 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.237223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.237199 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.288300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.288268 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99d4\" (UniqueName: \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\") pod \"maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.302277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.302248 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:02.689033 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:27:02.688968 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b41dab_d7e6_45b9_b6bc_98c08aeb7195.slice/crio-ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b WatchSource:0}: Error finding container ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b: Status 404 returned error can't find the container with id ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b Nov 04 01:27:02.690923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:02.690753 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p"] Nov 04 01:27:03.278358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.278318 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7"] Nov 04 01:27:03.283080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.283058 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.325954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.325922 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7"] Nov 04 01:27:03.355816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.355782 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.356067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.356037 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.356548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.356522 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7766\" (UniqueName: \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.458828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.458791 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.459068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.458836 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.459068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.459052 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7766\" (UniqueName: \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.459669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.459640 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.461961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.461935 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.514256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.514231 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7766\" (UniqueName: \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\") pod \"maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.543139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.543107 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" event={"ID":"41b41dab-d7e6-45b9-b6bc-98c08aeb7195","Type":"ContainerStarted","Data":"d72f5217c0a335e23ba960c416defea168cd11df8fb16e0aef3fda084707a365"} Nov 04 01:27:03.543329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.543171 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" event={"ID":"41b41dab-d7e6-45b9-b6bc-98c08aeb7195","Type":"ContainerStarted","Data":"ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b"} Nov 04 01:27:03.599199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.599140 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:03.661025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.660927 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" podStartSLOduration=2.660909615 podStartE2EDuration="2.660909615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:03.656745572 +0000 UTC m=+1792.415407736" watchObservedRunningTime="2026-11-04 01:27:03.660909615 +0000 UTC m=+1792.419571777" Nov 04 01:27:03.999163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:03.999134 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7"] Nov 04 01:27:04.556911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:04.556810 4486 generic.go:356] "Generic (PLEG): container finished" podID="41b41dab-d7e6-45b9-b6bc-98c08aeb7195" containerID="d72f5217c0a335e23ba960c416defea168cd11df8fb16e0aef3fda084707a365" exitCode=0 Nov 04 01:27:04.556911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:04.556880 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" event={"ID":"41b41dab-d7e6-45b9-b6bc-98c08aeb7195","Type":"ContainerDied","Data":"d72f5217c0a335e23ba960c416defea168cd11df8fb16e0aef3fda084707a365"} Nov 04 01:27:04.558967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:04.558938 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" event={"ID":"e8883bf0-c41c-45c6-8e24-82e2ced9aca4","Type":"ContainerStarted","Data":"2c402e479cc2609b40d62860f3eeda0eb3886c885b20609f2b4e1e0206628cf2"} Nov 04 01:27:04.559157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:04.558998 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" event={"ID":"e8883bf0-c41c-45c6-8e24-82e2ced9aca4","Type":"ContainerStarted","Data":"832830f0747c682c981eceddb451cf8bf04e410fe548e00093b588e67e4abd8d"} Nov 04 01:27:04.792420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:04.792361 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" podStartSLOduration=1.792344683 podStartE2EDuration="1.792344683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:04.788795207 +0000 UTC m=+1793.547457370" watchObservedRunningTime="2026-11-04 01:27:04.792344683 +0000 UTC m=+1793.551006846" Nov 04 01:27:05.575746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.575702 4486 generic.go:356] "Generic (PLEG): container finished" podID="e8883bf0-c41c-45c6-8e24-82e2ced9aca4" containerID="2c402e479cc2609b40d62860f3eeda0eb3886c885b20609f2b4e1e0206628cf2" exitCode=0 Nov 04 01:27:05.576284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.575827 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" event={"ID":"e8883bf0-c41c-45c6-8e24-82e2ced9aca4","Type":"ContainerDied","Data":"2c402e479cc2609b40d62860f3eeda0eb3886c885b20609f2b4e1e0206628cf2"} Nov 04 01:27:05.767586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.767556 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:05.903363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.903273 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\" (UniqueName: \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\") pod \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " Nov 04 01:27:05.903363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.903357 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\") pod \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " Nov 04 01:27:05.903601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.903497 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\") pod \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\" (UID: \"41b41dab-d7e6-45b9-b6bc-98c08aeb7195\") " Nov 04 01:27:05.904192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.904161 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca" pod "41b41dab-d7e6-45b9-b6bc-98c08aeb7195" (UID: "41b41dab-d7e6-45b9-b6bc-98c08aeb7195"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:05.904850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.904813 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:05.907101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.907062 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca" pod "41b41dab-d7e6-45b9-b6bc-98c08aeb7195" (UID: "41b41dab-d7e6-45b9-b6bc-98c08aeb7195"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:05.907391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:05.907365 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4" pod "41b41dab-d7e6-45b9-b6bc-98c08aeb7195" (UID: "41b41dab-d7e6-45b9-b6bc-98c08aeb7195"). InnerVolumeSpecName "kube-api-access-t99d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:06.006797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.006764 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99d4\" (UniqueName: \"kubernetes.io/projected/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-kube-api-access-t99d4\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:06.006797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.006792 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b41dab-d7e6-45b9-b6bc-98c08aeb7195-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:06.590775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.590746 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" Nov 04 01:27:06.591256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.590778 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p" event={"ID":"41b41dab-d7e6-45b9-b6bc-98c08aeb7195","Type":"ContainerDied","Data":"ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b"} Nov 04 01:27:06.591256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.590837 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac65dfb52c6a17b9553d9b60deb58d01a7fa4cc004342b3821a7204203824f8b" Nov 04 01:27:06.776098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.776071 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:06.922225 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.922140 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\" (UniqueName: \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\") pod \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " Nov 04 01:27:06.922399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.922226 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\") pod \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " Nov 04 01:27:06.922399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.922302 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\") pod \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\" (UID: \"e8883bf0-c41c-45c6-8e24-82e2ced9aca4\") " Nov 04 01:27:06.922943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.922905 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca" pod "e8883bf0-c41c-45c6-8e24-82e2ced9aca4" (UID: "e8883bf0-c41c-45c6-8e24-82e2ced9aca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:06.923166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.923137 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:06.924424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.924402 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca" pod "e8883bf0-c41c-45c6-8e24-82e2ced9aca4" (UID: "e8883bf0-c41c-45c6-8e24-82e2ced9aca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:06.924870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:06.924851 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766" pod "e8883bf0-c41c-45c6-8e24-82e2ced9aca4" (UID: "e8883bf0-c41c-45c6-8e24-82e2ced9aca4"). InnerVolumeSpecName "kube-api-access-d7766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:07.024486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.024447 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7766\" (UniqueName: \"kubernetes.io/projected/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-kube-api-access-d7766\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:07.024486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.024479 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8883bf0-c41c-45c6-8e24-82e2ced9aca4-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:07.601130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.601100 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" Nov 04 01:27:07.601130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.601123 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7" event={"ID":"e8883bf0-c41c-45c6-8e24-82e2ced9aca4","Type":"ContainerDied","Data":"832830f0747c682c981eceddb451cf8bf04e410fe548e00093b588e67e4abd8d"} Nov 04 01:27:07.601669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.601168 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="832830f0747c682c981eceddb451cf8bf04e410fe548e00093b588e67e4abd8d" Nov 04 01:27:07.708087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:07.708006 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:27:12.073134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.073096 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj"] Nov 04 01:27:12.073838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.073823 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e8883bf0-c41c-45c6-8e24-82e2ced9aca4" containerName="revoke-keys" Nov 04 01:27:12.073886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.073838 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8883bf0-c41c-45c6-8e24-82e2ced9aca4" containerName="revoke-keys" Nov 04 01:27:12.073886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.073859 4486 cpu_manager.go:414] "containerMap: removing container" podUID="41b41dab-d7e6-45b9-b6bc-98c08aeb7195" containerName="revoke-keys" Nov 04 01:27:12.073886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.073864 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b41dab-d7e6-45b9-b6bc-98c08aeb7195" containerName="revoke-keys" Nov 04 01:27:12.074015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.074005 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8883bf0-c41c-45c6-8e24-82e2ced9aca4" containerName="revoke-keys" Nov 04 01:27:12.074050 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.074017 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b41dab-d7e6-45b9-b6bc-98c08aeb7195" containerName="revoke-keys" Nov 04 01:27:12.077959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.077940 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.162528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.157057 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj"] Nov 04 01:27:12.171897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.171864 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:27:12.172164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.171917 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:27:12.179024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.178974 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.179243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.179202 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdkr\" (UniqueName: \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.179377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.179353 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.281441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.281397 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.281643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.281553 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.281736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.281720 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdkr\" (UniqueName: \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.282429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.282397 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.284347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.284320 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.349059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.339859 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdkr\" (UniqueName: \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\") pod \"maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.392873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.392844 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:12.808156 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:27:12.808124 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ec5253_5e32_4910_933b_f2543d722b78.slice/crio-f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5 WatchSource:0}: Error finding container f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5: Status 404 returned error can't find the container with id f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5 Nov 04 01:27:12.809385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:12.809349 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj"] Nov 04 01:27:13.693232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:13.693194 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" event={"ID":"52ec5253-5e32-4910-933b-f2543d722b78","Type":"ContainerStarted","Data":"27e7cdd9fad17b23666ccd65d8b929bca6140efc3d90ef0fe7ced42006e7860d"} Nov 04 01:27:13.693649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:13.693248 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" event={"ID":"52ec5253-5e32-4910-933b-f2543d722b78","Type":"ContainerStarted","Data":"f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5"} Nov 04 01:27:13.878516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:13.878446 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" podStartSLOduration=1.878424591 podStartE2EDuration="1.878424591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:13.805465978 +0000 UTC m=+1802.564128142" watchObservedRunningTime="2026-11-04 01:27:13.878424591 +0000 UTC m=+1802.637086753" Nov 04 01:27:13.879043 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:27:13.878957 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:27:14.772721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:14.765253 4486 generic.go:356] "Generic (PLEG): container finished" podID="52ec5253-5e32-4910-933b-f2543d722b78" containerID="27e7cdd9fad17b23666ccd65d8b929bca6140efc3d90ef0fe7ced42006e7860d" exitCode=0 Nov 04 01:27:14.772721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:14.765426 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" event={"ID":"52ec5253-5e32-4910-933b-f2543d722b78","Type":"ContainerDied","Data":"27e7cdd9fad17b23666ccd65d8b929bca6140efc3d90ef0fe7ced42006e7860d"} Nov 04 01:27:15.105008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:15.103632 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc"] Nov 04 01:27:15.121767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:15.121732 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4416988-6v4hc"] Nov 04 01:27:15.375836 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:15.375757 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a4fa9a-7b83-4040-ac51-a482ae0a5e89" path="/var/lib/kubelet/pods/10a4fa9a-7b83-4040-ac51-a482ae0a5e89/volumes" Nov 04 01:27:16.015401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.015369 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:16.152251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.152219 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\") pod \"52ec5253-5e32-4910-933b-f2543d722b78\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " Nov 04 01:27:16.152463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.152444 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\" (UniqueName: \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\") pod \"52ec5253-5e32-4910-933b-f2543d722b78\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " Nov 04 01:27:16.152612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.152598 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\") pod \"52ec5253-5e32-4910-933b-f2543d722b78\" (UID: \"52ec5253-5e32-4910-933b-f2543d722b78\") " Nov 04 01:27:16.153143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.153114 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca" pod "52ec5253-5e32-4910-933b-f2543d722b78" (UID: "52ec5253-5e32-4910-933b-f2543d722b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:16.153533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.153511 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.154589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.154555 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca" pod "52ec5253-5e32-4910-933b-f2543d722b78" (UID: "52ec5253-5e32-4910-933b-f2543d722b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:16.155317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.155294 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr" pod "52ec5253-5e32-4910-933b-f2543d722b78" (UID: "52ec5253-5e32-4910-933b-f2543d722b78"). InnerVolumeSpecName "kube-api-access-6cdkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:16.255203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.255163 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ec5253-5e32-4910-933b-f2543d722b78-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.255203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.255199 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdkr\" (UniqueName: \"kubernetes.io/projected/52ec5253-5e32-4910-933b-f2543d722b78-kube-api-access-6cdkr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:16.805702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.805669 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" Nov 04 01:27:16.805890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.805710 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj" event={"ID":"52ec5253-5e32-4910-933b-f2543d722b78","Type":"ContainerDied","Data":"f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5"} Nov 04 01:27:16.805890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:16.805761 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f538f5fa4cad5d839b566973e8cb0e10d34b4ca9e8672a0854a5b56e24a9a7f5" Nov 04 01:27:17.032595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.032561 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv"] Nov 04 01:27:17.033427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.033407 4486 cpu_manager.go:414] "containerMap: removing container" podUID="52ec5253-5e32-4910-933b-f2543d722b78" containerName="revoke-keys" Nov 04 01:27:17.033478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.033427 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ec5253-5e32-4910-933b-f2543d722b78" containerName="revoke-keys" Nov 04 01:27:17.033569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.033559 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ec5253-5e32-4910-933b-f2543d722b78" containerName="revoke-keys" Nov 04 01:27:17.037303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.037280 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.088598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.088221 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:27:17.088825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.088798 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:27:17.091640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.091614 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv"] Nov 04 01:27:17.170505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.170470 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.170764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.170738 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkdq\" (UniqueName: \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.171031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.170958 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.272361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.272327 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.272523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.272491 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkdq\" (UniqueName: \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.272648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.272633 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.273262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.273231 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.275612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.275579 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.328871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.328835 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkdq\" (UniqueName: \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\") pod \"maas-api-revoke-sub-33e734d5-bz2fv\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.356442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.355895 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:17.769066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.768081 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:27:17.840887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.840855 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv"] Nov 04 01:27:17.859535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.852638 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:27:17.906652 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.906528 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:27:17.911750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.911720 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:27:17.944052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:17.936681 4486 kubelet.go:1689] "Image garbage collection succeeded" Nov 04 01:27:18.701952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.701920 4486 scope.go:122] "RemoveContainer" containerID="692d7e772b6babe5c65e188cd529331d5bd170b6b4b2401a7ce33f5e5d1c3c96" Nov 04 01:27:18.714052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.714031 4486 scope.go:122] "RemoveContainer" containerID="3dba2746186e0892a78f33f7c7f0c488938ef233925071c743919a79ca1d2932" Nov 04 01:27:18.724694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.724666 4486 scope.go:122] "RemoveContainer" containerID="edf755f6e9ea9ac709ab39dc8ff2371b6b7b2b56259046201130d0a1e73dfc9a" Nov 04 01:27:18.734929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.734904 4486 scope.go:122] "RemoveContainer" containerID="0bbf578e064b206c44c84f3f23ce7cd02bc57f644162e1f367f1c43bdabf3148" Nov 04 01:27:18.745685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.745660 4486 scope.go:122] "RemoveContainer" containerID="2a31365e04a577fb285edc23cf2dac2c08118949bbc7407bfb40390f094f17e7" Nov 04 01:27:18.755834 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.755814 4486 scope.go:122] "RemoveContainer" containerID="fc41be01778e51dd15e6406a798dbc1317fc63dabf75b908516ec08c74240465" Nov 04 01:27:18.768714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.768691 4486 scope.go:122] "RemoveContainer" containerID="9c634da9068349f67064eac132313ad9d62a09c45c4457b59cc032eb85b0b00d" Nov 04 01:27:18.778915 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.778885 4486 scope.go:122] "RemoveContainer" containerID="3532843b3f19a58c05a381fd7e327cf619de944c850ff63d084e96e4151b42ab" Nov 04 01:27:18.788741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.788723 4486 scope.go:122] "RemoveContainer" containerID="bfa8ba4a6f52d695271a4df1803c6dd493f93b396d5a64850be817a76df2b8a5" Nov 04 01:27:18.859003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.858179 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" event={"ID":"0cbb94a6-08da-402b-93ab-e3a1d4608816","Type":"ContainerStarted","Data":"fe75e7e379b2cad32143958f477fd7802cbc2ff76c4f1587ea1b7afcf3f319a6"} Nov 04 01:27:18.859003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.858272 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" event={"ID":"0cbb94a6-08da-402b-93ab-e3a1d4608816","Type":"ContainerStarted","Data":"21a1fdab24bb315bf872e704fb72c56f5cc267b83ba94d1220f8e3a3f6b36fdb"} Nov 04 01:27:18.985014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:18.984947 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" podStartSLOduration=2.9849276319999998 podStartE2EDuration="2.984927632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:18.983056047 +0000 UTC m=+1807.741718210" watchObservedRunningTime="2026-11-04 01:27:18.984927632 +0000 UTC m=+1807.743589794" Nov 04 01:27:19.881639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:19.881607 4486 generic.go:356] "Generic (PLEG): container finished" podID="0cbb94a6-08da-402b-93ab-e3a1d4608816" containerID="fe75e7e379b2cad32143958f477fd7802cbc2ff76c4f1587ea1b7afcf3f319a6" exitCode=0 Nov 04 01:27:19.882072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:19.881693 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" event={"ID":"0cbb94a6-08da-402b-93ab-e3a1d4608816","Type":"ContainerDied","Data":"fe75e7e379b2cad32143958f477fd7802cbc2ff76c4f1587ea1b7afcf3f319a6"} Nov 04 01:27:20.060009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:20.056401 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6"] Nov 04 01:27:20.078131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:20.070620 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d71be16-g59p6"] Nov 04 01:27:21.160400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.160340 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:21.179358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.179331 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\") pod \"0cbb94a6-08da-402b-93ab-e3a1d4608816\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " Nov 04 01:27:21.179510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.179381 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\" (UniqueName: \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\") pod \"0cbb94a6-08da-402b-93ab-e3a1d4608816\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " Nov 04 01:27:21.179644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.179624 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\") pod \"0cbb94a6-08da-402b-93ab-e3a1d4608816\" (UID: \"0cbb94a6-08da-402b-93ab-e3a1d4608816\") " Nov 04 01:27:21.179749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.179723 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca" pod "0cbb94a6-08da-402b-93ab-e3a1d4608816" (UID: "0cbb94a6-08da-402b-93ab-e3a1d4608816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:21.180333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.180313 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:21.189535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.189512 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca" pod "0cbb94a6-08da-402b-93ab-e3a1d4608816" (UID: "0cbb94a6-08da-402b-93ab-e3a1d4608816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:21.189668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.189652 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq" pod "0cbb94a6-08da-402b-93ab-e3a1d4608816" (UID: "0cbb94a6-08da-402b-93ab-e3a1d4608816"). InnerVolumeSpecName "kube-api-access-slkdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:21.282685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.282649 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkdq\" (UniqueName: \"kubernetes.io/projected/0cbb94a6-08da-402b-93ab-e3a1d4608816-kube-api-access-slkdq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:21.282685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.282688 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbb94a6-08da-402b-93ab-e3a1d4608816-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:21.353794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.353756 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be675f53-aeb3-4c7e-9e0c-52b247f54ce3" path="/var/lib/kubelet/pods/be675f53-aeb3-4c7e-9e0c-52b247f54ce3/volumes" Nov 04 01:27:21.907310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.907281 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" Nov 04 01:27:21.907310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.907302 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv" event={"ID":"0cbb94a6-08da-402b-93ab-e3a1d4608816","Type":"ContainerDied","Data":"21a1fdab24bb315bf872e704fb72c56f5cc267b83ba94d1220f8e3a3f6b36fdb"} Nov 04 01:27:21.907575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:21.907359 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a1fdab24bb315bf872e704fb72c56f5cc267b83ba94d1220f8e3a3f6b36fdb" Nov 04 01:27:22.754359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:22.754286 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" containerID="cri-o://4975df01c498c89ba5796ef032b37b94ff01a08eb0f626e1da5b668ec742f89c" gracePeriod=45 Nov 04 01:27:22.806541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:22.806497 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused" Nov 04 01:27:22.921339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:22.921304 4486 generic.go:356] "Generic (PLEG): container finished" podID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerID="4975df01c498c89ba5796ef032b37b94ff01a08eb0f626e1da5b668ec742f89c" exitCode=0 Nov 04 01:27:22.921510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:22.921373 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" event={"ID":"829b4a50-3c8a-4531-924a-837c0b9bac8b","Type":"ContainerDied","Data":"4975df01c498c89ba5796ef032b37b94ff01a08eb0f626e1da5b668ec742f89c"} Nov 04 01:27:23.087826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.087802 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:27:23.111373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111343 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\" (UniqueName: \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.111562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111381 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.111562 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111453 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.111693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111639 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.111944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111921 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.112031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.111971 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\") pod \"829b4a50-3c8a-4531-924a-837c0b9bac8b\" (UID: \"829b4a50-3c8a-4531-924a-837c0b9bac8b\") " Nov 04 01:27:23.112259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.112235 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.112430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.112412 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.112874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.112852 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.112874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.112875 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.113281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.113140 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.115874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.115848 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:27:23.116414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.116393 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "kube-api-access-d4qk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:23.116665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.116644 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs" pod "829b4a50-3c8a-4531-924a-837c0b9bac8b" (UID: "829b4a50-3c8a-4531-924a-837c0b9bac8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:23.216130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.216096 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.216373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.216353 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829b4a50-3c8a-4531-924a-837c0b9bac8b-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.216520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.216477 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qk5\" (UniqueName: \"kubernetes.io/projected/829b4a50-3c8a-4531-924a-837c0b9bac8b-kube-api-access-d4qk5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.216520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.216501 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829b4a50-3c8a-4531-924a-837c0b9bac8b-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:23.934260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.934227 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" Nov 04 01:27:23.934831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.934268 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2" event={"ID":"829b4a50-3c8a-4531-924a-837c0b9bac8b","Type":"ContainerDied","Data":"e49817c8175c6543e358f76569c5481cb4bf9e9137ec73d0c581d40adfe3acc2"} Nov 04 01:27:23.934831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:23.934329 4486 scope.go:122] "RemoveContainer" containerID="4975df01c498c89ba5796ef032b37b94ff01a08eb0f626e1da5b668ec742f89c" Nov 04 01:27:24.098064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.098029 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:27:24.146937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.146898 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2ee045/premium-w3-2ee045-kserve-5885fb4578-ldkh2"] Nov 04 01:27:24.533823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.533791 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:24.534327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.533853 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:24.696612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.696529 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh"] Nov 04 01:27:24.697458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697435 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0cbb94a6-08da-402b-93ab-e3a1d4608816" containerName="revoke-keys" Nov 04 01:27:24.697458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697457 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbb94a6-08da-402b-93ab-e3a1d4608816" containerName="revoke-keys" Nov 04 01:27:24.697645 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697523 4486 cpu_manager.go:414] "containerMap: removing container" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" Nov 04 01:27:24.697645 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697531 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" Nov 04 01:27:24.697762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697746 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbb94a6-08da-402b-93ab-e3a1d4608816" containerName="revoke-keys" Nov 04 01:27:24.697817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.697775 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" containerName="main" Nov 04 01:27:24.700473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.700451 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.745771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.745722 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.745948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.745887 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.746145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.746126 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7jw\" (UniqueName: \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.765406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.765366 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:27:24.766332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.766307 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh"] Nov 04 01:27:24.782378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.782346 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p"] Nov 04 01:27:24.787297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.787274 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.804928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.804906 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:27:24.821506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.821480 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:27:24.832454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.832425 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p"] Nov 04 01:27:24.847726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.847694 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.848002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.847778 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.848002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.847836 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.848002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.847953 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.848207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.848035 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfsk\" (UniqueName: \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.848207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.848126 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7jw\" (UniqueName: \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.848646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.848619 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.851485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.851437 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.904102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.904070 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7jw\" (UniqueName: \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\") pod \"maas-api-revoke-sub-b7bd7e9a-vt2xh\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:24.947286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.947199 4486 generic.go:356] "Generic (PLEG): container finished" podID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerID="fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285" exitCode=0 Nov 04 01:27:24.947286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.947271 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerDied","Data":"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285"} Nov 04 01:27:24.947864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.947845 4486 scope.go:122] "RemoveContainer" containerID="fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285" Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.949648 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.949723 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfsk\" (UniqueName: \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.950028 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.950184 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.950264 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.950336 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:24.951009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.950842 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:24.952957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:24.952933 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:25.005606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.005578 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfsk\" (UniqueName: \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\") pod \"maas-api-revoke-sub-d40c53fb-7qc7p\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:25.014574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.014549 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:25.103558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.103279 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:25.394005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.393910 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829b4a50-3c8a-4531-924a-837c0b9bac8b" path="/var/lib/kubelet/pods/829b4a50-3c8a-4531-924a-837c0b9bac8b/volumes" Nov 04 01:27:25.499070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.498955 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh"] Nov 04 01:27:25.582502 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:27:25.582466 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8439e5_a2b1_42b4_92de_0c246986a1ab.slice/crio-bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b WatchSource:0}: Error finding container bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b: Status 404 returned error can't find the container with id bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b Nov 04 01:27:25.583184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:25.583147 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p"] Nov 04 01:27:26.007079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.006944 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerStarted","Data":"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a"} Nov 04 01:27:26.008720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.008694 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" event={"ID":"e41837e1-c491-4797-93b4-83213177db4a","Type":"ContainerStarted","Data":"13ec2e97a9b946f183c4dc813ee733badb275db522ff3d3e711b98dac48059da"} Nov 04 01:27:26.010105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.010083 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" event={"ID":"8f8439e5-a2b1-42b4-92de-0c246986a1ab","Type":"ContainerStarted","Data":"bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b"} Nov 04 01:27:26.833013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.832368 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:27:26.833013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.832771 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:26.833013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:26.832822 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:27.099408 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.099370 4486 generic.go:356] "Generic (PLEG): container finished" podID="e41837e1-c491-4797-93b4-83213177db4a" containerID="ea900b1f1c048738430bdfd7fd4ffa2d9e548e06020e3eab09182e82eeafa530" exitCode=0 Nov 04 01:27:27.100145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.100114 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" event={"ID":"e41837e1-c491-4797-93b4-83213177db4a","Type":"ContainerDied","Data":"ea900b1f1c048738430bdfd7fd4ffa2d9e548e06020e3eab09182e82eeafa530"} Nov 04 01:27:27.124009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.116748 4486 generic.go:356] "Generic (PLEG): container finished" podID="8f8439e5-a2b1-42b4-92de-0c246986a1ab" containerID="4fae13aa8890db2144d3c7561879a6ace380248f0d150523f1e82500fd21a732" exitCode=0 Nov 04 01:27:27.124009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.118025 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" event={"ID":"8f8439e5-a2b1-42b4-92de-0c246986a1ab","Type":"ContainerDied","Data":"4fae13aa8890db2144d3c7561879a6ace380248f0d150523f1e82500fd21a732"} Nov 04 01:27:27.836008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.832330 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:27.836008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:27.832393 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:28.600949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.600205 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:28.608441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.608415 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:28.692420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.692386 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\" (UniqueName: \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\") pod \"e41837e1-c491-4797-93b4-83213177db4a\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " Nov 04 01:27:28.692618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.692510 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\") pod \"e41837e1-c491-4797-93b4-83213177db4a\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " Nov 04 01:27:28.692689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.692626 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\") pod \"e41837e1-c491-4797-93b4-83213177db4a\" (UID: \"e41837e1-c491-4797-93b4-83213177db4a\") " Nov 04 01:27:28.692689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.692661 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\") pod \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " Nov 04 01:27:28.693586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.693559 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca" pod "e41837e1-c491-4797-93b4-83213177db4a" (UID: "e41837e1-c491-4797-93b4-83213177db4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:28.695512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.695478 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca" pod "e41837e1-c491-4797-93b4-83213177db4a" (UID: "e41837e1-c491-4797-93b4-83213177db4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:28.695626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.695551 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca" pod "8f8439e5-a2b1-42b4-92de-0c246986a1ab" (UID: "8f8439e5-a2b1-42b4-92de-0c246986a1ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:28.695717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.695699 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw" pod "e41837e1-c491-4797-93b4-83213177db4a" (UID: "e41837e1-c491-4797-93b4-83213177db4a"). InnerVolumeSpecName "kube-api-access-ch7jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:28.794257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.794162 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\") pod \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " Nov 04 01:27:28.794432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.794297 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\" (UniqueName: \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\") pod \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\" (UID: \"8f8439e5-a2b1-42b4-92de-0c246986a1ab\") " Nov 04 01:27:28.794581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.794556 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca" pod "8f8439e5-a2b1-42b4-92de-0c246986a1ab" (UID: "8f8439e5-a2b1-42b4-92de-0c246986a1ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:28.795165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.795142 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7jw\" (UniqueName: \"kubernetes.io/projected/e41837e1-c491-4797-93b4-83213177db4a-kube-api-access-ch7jw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:28.795165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.795165 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e41837e1-c491-4797-93b4-83213177db4a-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:28.795324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.795176 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e41837e1-c491-4797-93b4-83213177db4a-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:28.795324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.795185 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:28.795324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.795194 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8439e5-a2b1-42b4-92de-0c246986a1ab-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:28.796418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.796393 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk" pod "8f8439e5-a2b1-42b4-92de-0c246986a1ab" (UID: "8f8439e5-a2b1-42b4-92de-0c246986a1ab"). InnerVolumeSpecName "kube-api-access-xzfsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:28.881260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.881222 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:28.881453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.881280 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:28.897433 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:28.897346 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzfsk\" (UniqueName: \"kubernetes.io/projected/8f8439e5-a2b1-42b4-92de-0c246986a1ab-kube-api-access-xzfsk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:29.172704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.172661 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" event={"ID":"e41837e1-c491-4797-93b4-83213177db4a","Type":"ContainerDied","Data":"13ec2e97a9b946f183c4dc813ee733badb275db522ff3d3e711b98dac48059da"} Nov 04 01:27:29.172879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.172738 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ec2e97a9b946f183c4dc813ee733badb275db522ff3d3e711b98dac48059da" Nov 04 01:27:29.172879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.172825 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh" Nov 04 01:27:29.174804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.174780 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" event={"ID":"8f8439e5-a2b1-42b4-92de-0c246986a1ab","Type":"ContainerDied","Data":"bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b"} Nov 04 01:27:29.174976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.174963 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7dd01e6542b887d5136c99aff3189b1f43dbed46485b5948686c4d385cb62b" Nov 04 01:27:29.175076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.174799 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p" Nov 04 01:27:29.823118 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.823063 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" containerID="cri-o://674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac" gracePeriod=25 Nov 04 01:27:29.831282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.831250 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:29.831521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:29.831474 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:30.861007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:30.860316 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 04 01:27:30.861007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:30.860391 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 04 01:27:31.860621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:31.859170 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:27:31.860621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:31.859748 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:27:31.862396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:31.862270 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:27:34.054094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:34.054056 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s"] Nov 04 01:27:34.101529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:34.101498 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c85061fb-gsv4s"] Nov 04 01:27:35.351677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:35.351642 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2" path="/var/lib/kubelet/pods/bef22b75-b5bb-4ea0-9680-e49aa3e2a8d2/volumes" Nov 04 01:27:37.790661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:37.790615 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:27:43.866225 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:27:43.866186 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:27:47.790287 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:47.790242 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:27:50.325161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.325132 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:27:50.394266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.394239 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\" (UniqueName: \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\") pod \"4039758e-0b4d-460d-b6bd-134b1dbc3764\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " Nov 04 01:27:50.406549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.394333 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\") pod \"4039758e-0b4d-460d-b6bd-134b1dbc3764\" (UID: \"4039758e-0b4d-460d-b6bd-134b1dbc3764\") " Nov 04 01:27:50.406549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.394655 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume" pod "4039758e-0b4d-460d-b6bd-134b1dbc3764" (UID: "4039758e-0b4d-460d-b6bd-134b1dbc3764"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:50.406549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.395265 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4039758e-0b4d-460d-b6bd-134b1dbc3764-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:50.406549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.396407 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp" pod "4039758e-0b4d-460d-b6bd-134b1dbc3764" (UID: "4039758e-0b4d-460d-b6bd-134b1dbc3764"). InnerVolumeSpecName "kube-api-access-8vwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:50.487454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.487418 4486 generic.go:356] "Generic (PLEG): container finished" podID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerID="674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac" exitCode=0 Nov 04 01:27:50.487616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.487479 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" Nov 04 01:27:50.487616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.487510 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" event={"ID":"4039758e-0b4d-460d-b6bd-134b1dbc3764","Type":"ContainerDied","Data":"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac"} Nov 04 01:27:50.487616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.487564 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9" event={"ID":"4039758e-0b4d-460d-b6bd-134b1dbc3764","Type":"ContainerDied","Data":"6f7a5b67d7b2df485b6d10509e03baab699569d14220e260e2d716ebf186a880"} Nov 04 01:27:50.487616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.487587 4486 scope.go:122] "RemoveContainer" containerID="674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac" Nov 04 01:27:50.496633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.496605 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwhp\" (UniqueName: \"kubernetes.io/projected/4039758e-0b4d-460d-b6bd-134b1dbc3764-kube-api-access-8vwhp\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:50.505651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.505622 4486 scope.go:122] "RemoveContainer" containerID="674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac" Nov 04 01:27:50.506040 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:27:50.505954 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac\": container with ID starting with 674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac not found: ID does not exist" containerID="674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac" Nov 04 01:27:50.506129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.506049 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac"} err="failed to get container status \"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac\": rpc error: code = NotFound desc = could not find container \"674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac\": container with ID starting with 674e1e4f351086182d9012f07045c4f9258d2d218394c949183d076f40e829ac not found: ID does not exist" Nov 04 01:27:50.661113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.661017 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:27:50.665851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:50.665826 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2ee045-858fd48cd-vjrq9"] Nov 04 01:27:51.351883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.351841 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" path="/var/lib/kubelet/pods/4039758e-0b4d-460d-b6bd-134b1dbc3764/volumes" Nov 04 01:27:51.498953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.498920 4486 generic.go:356] "Generic (PLEG): container finished" podID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerID="873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6" exitCode=0 Nov 04 01:27:51.499169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.499005 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerDied","Data":"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6"} Nov 04 01:27:51.499521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.499504 4486 scope.go:122] "RemoveContainer" containerID="873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6" Nov 04 01:27:51.501615 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.501581 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:51.501731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.501672 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:51.501731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:51.501718 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:27:52.510994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:52.510960 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerStarted","Data":"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9"} Nov 04 01:27:52.995930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:52.995831 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:27:52.997027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:52.996978 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 01:27:52.997174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:52.997048 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 01:27:53.860173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:53.860127 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:27:53.995626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:53.995592 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 01:27:53.995799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:53.995645 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 01:27:54.995779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:54.995740 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 01:27:54.996238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:54.995807 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 01:27:56.037006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:56.033609 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 01:27:56.037006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:56.033677 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 01:27:57.008004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:56.997279 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 04 01:27:57.008004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:56.997354 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 04 01:27:58.015947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:58.015892 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:27:58.019041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:58.019015 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:27:58.553191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:58.553159 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p"] Nov 04 01:27:58.558184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:58.558158 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f10575b3-2tt9p"] Nov 04 01:27:59.251142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.251101 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:27:59.252293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252268 4486 cpu_manager.go:414] "containerMap: removing container" podUID="8f8439e5-a2b1-42b4-92de-0c246986a1ab" containerName="revoke-keys" Nov 04 01:27:59.252348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252296 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8439e5-a2b1-42b4-92de-0c246986a1ab" containerName="revoke-keys" Nov 04 01:27:59.252398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252351 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e41837e1-c491-4797-93b4-83213177db4a" containerName="revoke-keys" Nov 04 01:27:59.252398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252359 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41837e1-c491-4797-93b4-83213177db4a" containerName="revoke-keys" Nov 04 01:27:59.252398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252391 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" Nov 04 01:27:59.252567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252399 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" Nov 04 01:27:59.252621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252610 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8439e5-a2b1-42b4-92de-0c246986a1ab" containerName="revoke-keys" Nov 04 01:27:59.252654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252631 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41837e1-c491-4797-93b4-83213177db4a" containerName="revoke-keys" Nov 04 01:27:59.252654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.252645 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4039758e-0b4d-460d-b6bd-134b1dbc3764" containerName="payload-processing" Nov 04 01:27:59.255894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.255871 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.310194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.310163 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-3dc6c8-openshift-default-dockercfg-cqfff\"" Nov 04 01:27:59.426473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.426395 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b41dab-d7e6-45b9-b6bc-98c08aeb7195" path="/var/lib/kubelet/pods/41b41dab-d7e6-45b9-b6bc-98c08aeb7195/volumes" Nov 04 01:27:59.427253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.427229 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460170 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460216 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460285 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460316 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlmx\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460420 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460497 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460588 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460647 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.461851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.460698 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567151 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567204 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567284 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567314 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlmx\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567410 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567480 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567568 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567622 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.567667 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.568601 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.579126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.568899 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.616860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.603240 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.616860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.603519 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.616860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.613601 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.623494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.623463 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.637167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.637127 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.658608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.658576 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlmx\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.663011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.662955 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\") pod \"e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:27:59.902786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:27:59.902439 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:28:00.449154 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:00.449106 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5224ab9_4358_481d_b5ba_c7603a5266fd.slice/crio-c132b5f1a0cfdd9a136719854360ca6321c6f3c2de7017484922867a62d1bda4 WatchSource:0}: Error finding container c132b5f1a0cfdd9a136719854360ca6321c6f3c2de7017484922867a62d1bda4: Status 404 returned error can't find the container with id c132b5f1a0cfdd9a136719854360ca6321c6f3c2de7017484922867a62d1bda4 Nov 04 01:28:00.449604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.449283 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:28:00.452603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.452563 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:00.452705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.452644 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:00.452705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.452682 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:00.687507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.687471 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerStarted","Data":"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564"} Nov 04 01:28:00.687679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.687526 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerStarted","Data":"c132b5f1a0cfdd9a136719854360ca6321c6f3c2de7017484922867a62d1bda4"} Nov 04 01:28:00.811309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.811198 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podStartSLOduration=1.8111788500000001 podStartE2EDuration="1.81117885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:00.803167946 +0000 UTC m=+1849.561830131" watchObservedRunningTime="2026-11-04 01:28:00.81117885 +0000 UTC m=+1849.569841012" Nov 04 01:28:00.902970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:00.902934 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:28:01.590330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.590300 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j"] Nov 04 01:28:01.593815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.593798 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.650173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.650138 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j"] Nov 04 01:28:01.668031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.663425 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxds\" (UniqueName: \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.668031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.663712 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.668031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.663869 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.689155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.689123 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:28:01.689352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.689336 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:28:01.765674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.765632 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.765853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.765790 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.766018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.765973 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxds\" (UniqueName: \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.766296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.766272 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.768208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.768188 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.820459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.820431 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxds\" (UniqueName: \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\") pod \"maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:01.908853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.908784 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:28:01.909024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:01.908887 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:02.291320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.291288 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j"] Nov 04 01:28:02.293242 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:02.293215 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b904f7_dbf3_4fc5_bc58_7fe693102882.slice/crio-bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2 WatchSource:0}: Error finding container bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2: Status 404 returned error can't find the container with id bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2 Nov 04 01:28:02.708746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.708712 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" event={"ID":"b2b904f7-dbf3-4fc5-bc58-7fe693102882","Type":"ContainerStarted","Data":"767699ea2474c1f8f321f75baded690b7bf88a4638f4f806ba789273e3ae914b"} Nov 04 01:28:02.709177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.708767 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" event={"ID":"b2b904f7-dbf3-4fc5-bc58-7fe693102882","Type":"ContainerStarted","Data":"bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2"} Nov 04 01:28:02.709177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.709158 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:28:02.710030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.709976 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:28:02.819385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:02.819331 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" podStartSLOduration=1.819317794 podStartE2EDuration="1.819317794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:02.816863015 +0000 UTC m=+1851.575525178" watchObservedRunningTime="2026-11-04 01:28:02.819317794 +0000 UTC m=+1851.577979954" Nov 04 01:28:03.725544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:03.725511 4486 generic.go:356] "Generic (PLEG): container finished" podID="b2b904f7-dbf3-4fc5-bc58-7fe693102882" containerID="767699ea2474c1f8f321f75baded690b7bf88a4638f4f806ba789273e3ae914b" exitCode=0 Nov 04 01:28:03.725940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:03.725595 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" event={"ID":"b2b904f7-dbf3-4fc5-bc58-7fe693102882","Type":"ContainerDied","Data":"767699ea2474c1f8f321f75baded690b7bf88a4638f4f806ba789273e3ae914b"} Nov 04 01:28:05.017779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.017754 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:05.077073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.077033 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\" (UniqueName: \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\") pod \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " Nov 04 01:28:05.077272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.077121 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\") pod \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " Nov 04 01:28:05.077272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.077198 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\") pod \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\" (UID: \"b2b904f7-dbf3-4fc5-bc58-7fe693102882\") " Nov 04 01:28:05.077526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.077490 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca" pod "b2b904f7-dbf3-4fc5-bc58-7fe693102882" (UID: "b2b904f7-dbf3-4fc5-bc58-7fe693102882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:05.078222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.078195 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.079466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.079431 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca" pod "b2b904f7-dbf3-4fc5-bc58-7fe693102882" (UID: "b2b904f7-dbf3-4fc5-bc58-7fe693102882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:05.079624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.079518 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds" pod "b2b904f7-dbf3-4fc5-bc58-7fe693102882" (UID: "b2b904f7-dbf3-4fc5-bc58-7fe693102882"). InnerVolumeSpecName "kube-api-access-7lxds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:05.179534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.179500 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxds\" (UniqueName: \"kubernetes.io/projected/b2b904f7-dbf3-4fc5-bc58-7fe693102882-kube-api-access-7lxds\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.179534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.179529 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b904f7-dbf3-4fc5-bc58-7fe693102882-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:05.778544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.778468 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" Nov 04 01:28:05.778544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.778498 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j" event={"ID":"b2b904f7-dbf3-4fc5-bc58-7fe693102882","Type":"ContainerDied","Data":"bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2"} Nov 04 01:28:05.778733 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:05.778568 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb85235fe3213a54ff583cc7b29ff33fb45cb446751068bf11314f7eda73ca2" Nov 04 01:28:06.073555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:06.073520 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:28:06.080263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:06.080231 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:28:06.159074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:06.158761 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj"] Nov 04 01:28:06.165321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:06.165295 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3d6ea20-5rqpj"] Nov 04 01:28:07.364543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.364504 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ec5253-5e32-4910-933b-f2543d722b78" path="/var/lib/kubelet/pods/52ec5253-5e32-4910-933b-f2543d722b78/volumes" Nov 04 01:28:07.581647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.581605 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:28:07.581835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.581804 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" containerID="cri-o://b07f3c45b17fc439bceae7930465deb0e6917a7519139369fbec1b6a2a3e1d14" gracePeriod=30 Nov 04 01:28:07.653063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.650067 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:28:07.653063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.650315 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" containerID="cri-o://295e505ebc89a9cacf37b0987d533b4f883f76fb1542a4bb6c556507822eeb19" gracePeriod=30 Nov 04 01:28:07.662159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.662088 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:28:07.662547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:07.662308 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" containerID="cri-o://e44f9604447ca9ec1005de36113ab3ade7ff2c92b277be2e66f2c2ad07a396fb" gracePeriod=30 Nov 04 01:28:09.291555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:09.291518 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:09.292093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:09.291579 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:12.877076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.875092 4486 generic.go:356] "Generic (PLEG): container finished" podID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerID="b07f3c45b17fc439bceae7930465deb0e6917a7519139369fbec1b6a2a3e1d14" exitCode=0 Nov 04 01:28:12.877076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.875325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" event={"ID":"2ec10c8c-b173-4cd3-8c66-b4a135253bef","Type":"ContainerDied","Data":"b07f3c45b17fc439bceae7930465deb0e6917a7519139369fbec1b6a2a3e1d14"} Nov 04 01:28:12.878444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.877781 4486 generic.go:356] "Generic (PLEG): container finished" podID="f6be3863-9fd0-4c96-837a-0d599552105e" containerID="e44f9604447ca9ec1005de36113ab3ade7ff2c92b277be2e66f2c2ad07a396fb" exitCode=0 Nov 04 01:28:12.878444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.878019 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" event={"ID":"f6be3863-9fd0-4c96-837a-0d599552105e","Type":"ContainerDied","Data":"e44f9604447ca9ec1005de36113ab3ade7ff2c92b277be2e66f2c2ad07a396fb"} Nov 04 01:28:12.880491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.880351 4486 generic.go:356] "Generic (PLEG): container finished" podID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerID="295e505ebc89a9cacf37b0987d533b4f883f76fb1542a4bb6c556507822eeb19" exitCode=0 Nov 04 01:28:12.880491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.880433 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" event={"ID":"46a8f3ad-f89b-4079-98b7-df4fc1b156e1","Type":"ContainerDied","Data":"295e505ebc89a9cacf37b0987d533b4f883f76fb1542a4bb6c556507822eeb19"} Nov 04 01:28:12.985902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:12.985870 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:28:13.064353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.064321 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:28:13.070619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.070589 4486 patch_prober.go:29] interesting pod/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Nov 04 01:28:13.070800 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.070638 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Nov 04 01:28:13.086491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086455 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.086721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086700 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.086799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086722 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.086799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086788 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.086918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086902 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.086975 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087059 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087137 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087273 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.087374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087315 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087388 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.087440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087425 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\") pod \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\" (UID: \"2ec10c8c-b173-4cd3-8c66-b4a135253bef\") " Nov 04 01:28:13.087665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087645 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.087785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.087683 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:13.088299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.088272 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.088399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.088301 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.088399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.088317 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.088399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.088333 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.088399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.088347 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.089155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.089131 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.089495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.089468 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "kube-api-access-chspz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.089644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.089622 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:13.089722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.089620 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy" pod "2ec10c8c-b173-4cd3-8c66-b4a135253bef" (UID: "2ec10c8c-b173-4cd3-8c66-b4a135253bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.189842 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.189732 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.189873 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.189998 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190068 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.190318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190132 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190164 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190201 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190287 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190360 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.190581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.190449 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\") pod \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\" (UID: \"46a8f3ad-f89b-4079-98b7-df4fc1b156e1\") " Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191477 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191552 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chspz\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-kube-api-access-chspz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191579 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191595 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191609 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec10c8c-b173-4cd3-8c66-b4a135253bef-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191623 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.191901 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.192077 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.192212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.192174 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:13.192781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.192683 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.193067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.193042 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:13.194403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.194366 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "kube-api-access-49hhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.194939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.194917 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy" pod "46a8f3ad-f89b-4079-98b7-df4fc1b156e1" (UID: "46a8f3ad-f89b-4079-98b7-df4fc1b156e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.294062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294020 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294066 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294083 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hhm\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-kube-api-access-49hhm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294097 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294111 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294124 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294138 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.294372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.294152 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8f3ad-f89b-4079-98b7-df4fc1b156e1-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.526204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.526177 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:28:13.613191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.613146 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:13.613448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.613203 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:13.699490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699441 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699525 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699671 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699720 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699797 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.699862 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700007 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700090 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700169 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.700228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700209 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\") pod \"f6be3863-9fd0-4c96-837a-0d599552105e\" (UID: \"f6be3863-9fd0-4c96-837a-0d599552105e\") " Nov 04 01:28:13.701038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700425 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.701038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.700909 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.701653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.701627 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.701792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.701659 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.701792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.701675 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.701961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.701835 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:13.701961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.701891 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.704897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.704872 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:13.705148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.705121 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.705446 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.705413 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:13.706564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.706537 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx" pod "f6be3863-9fd0-4c96-837a-0d599552105e" (UID: "f6be3863-9fd0-4c96-837a-0d599552105e"). InnerVolumeSpecName "kube-api-access-5vfbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:13.803597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803500 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6be3863-9fd0-4c96-837a-0d599552105e-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.803597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803544 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.803597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803560 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6be3863-9fd0-4c96-837a-0d599552105e-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.803597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803574 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.803597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803588 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6be3863-9fd0-4c96-837a-0d599552105e-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.804107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.803602 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfbx\" (UniqueName: \"kubernetes.io/projected/f6be3863-9fd0-4c96-837a-0d599552105e-kube-api-access-5vfbx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:13.857803 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:13.857761 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:28:13.890930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.890901 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" Nov 04 01:28:13.891499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.890938 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s" event={"ID":"2ec10c8c-b173-4cd3-8c66-b4a135253bef","Type":"ContainerDied","Data":"81b87964e3f7a36c5c1e0c0bf54e562d3879c747ba7ce64ca5c574ecdd77aa7f"} Nov 04 01:28:13.891499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.891017 4486 scope.go:122] "RemoveContainer" containerID="b07f3c45b17fc439bceae7930465deb0e6917a7519139369fbec1b6a2a3e1d14" Nov 04 01:28:13.892940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.892920 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" Nov 04 01:28:13.893104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.892941 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl" event={"ID":"f6be3863-9fd0-4c96-837a-0d599552105e","Type":"ContainerDied","Data":"cb78b4be1a7edf6feb424616c348e36cd80391f8711cf92fd202898b44ded058"} Nov 04 01:28:13.896151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.896126 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" event={"ID":"46a8f3ad-f89b-4079-98b7-df4fc1b156e1","Type":"ContainerDied","Data":"21ca26f04603c2fe5e912d92d692aa0f758dbf8726f395a5d1736042035e4cbe"} Nov 04 01:28:13.896279 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.896229 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv" Nov 04 01:28:13.911042 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.911018 4486 scope.go:122] "RemoveContainer" containerID="e44f9604447ca9ec1005de36113ab3ade7ff2c92b277be2e66f2c2ad07a396fb" Nov 04 01:28:13.929561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:13.929507 4486 scope.go:122] "RemoveContainer" containerID="295e505ebc89a9cacf37b0987d533b4f883f76fb1542a4bb6c556507822eeb19" Nov 04 01:28:14.060900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.060798 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:28:14.068691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.068654 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-lpx6s"] Nov 04 01:28:14.190379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.190331 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:28:14.287602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.282608 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-cqwsv"] Nov 04 01:28:14.349700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.349667 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:28:14.359034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.359005 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-wr6vl"] Nov 04 01:28:14.908690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.908595 4486 generic.go:356] "Generic (PLEG): container finished" podID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerID="73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95" exitCode=0 Nov 04 01:28:14.908690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.908670 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerDied","Data":"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95"} Nov 04 01:28:14.909301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.909186 4486 scope.go:122] "RemoveContainer" containerID="73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95" Nov 04 01:28:14.911271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.911197 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:14.911432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.911357 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:14.911524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:14.911505 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:15.357691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:15.357650 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" path="/var/lib/kubelet/pods/2ec10c8c-b173-4cd3-8c66-b4a135253bef/volumes" Nov 04 01:28:15.358217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:15.358199 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" path="/var/lib/kubelet/pods/46a8f3ad-f89b-4079-98b7-df4fc1b156e1/volumes" Nov 04 01:28:15.358671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:15.358653 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" path="/var/lib/kubelet/pods/f6be3863-9fd0-4c96-837a-0d599552105e/volumes" Nov 04 01:28:15.922810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:15.922773 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerStarted","Data":"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07"} Nov 04 01:28:16.177493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:16.177370 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:28:16.177854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:16.177829 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:16.201774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:16.178007 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:17.195009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:17.192154 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:17.195009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:17.192216 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:18.176633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.176596 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:18.176821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.176674 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:18.614089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614058 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl"] Nov 04 01:28:18.614829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614810 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" Nov 04 01:28:18.614829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614827 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614838 4486 cpu_manager.go:414] "containerMap: removing container" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614847 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614868 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614874 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614901 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b2b904f7-dbf3-4fc5-bc58-7fe693102882" containerName="revoke-keys" Nov 04 01:28:18.615020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.614906 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b904f7-dbf3-4fc5-bc58-7fe693102882" containerName="revoke-keys" Nov 04 01:28:18.615216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.615035 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6be3863-9fd0-4c96-837a-0d599552105e" containerName="istio-proxy" Nov 04 01:28:18.615216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.615050 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b904f7-dbf3-4fc5-bc58-7fe693102882" containerName="revoke-keys" Nov 04 01:28:18.615216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.615061 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec10c8c-b173-4cd3-8c66-b4a135253bef" containerName="istio-proxy" Nov 04 01:28:18.615216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.615067 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a8f3ad-f89b-4079-98b7-df4fc1b156e1" containerName="istio-proxy" Nov 04 01:28:18.618936 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.618918 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.672714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.672676 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl"] Nov 04 01:28:18.695028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.693881 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:28:18.695028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.693917 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:28:18.769947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.769902 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.770202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.770061 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82nb\" (UniqueName: \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.770202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.770109 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.871695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.871600 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.871876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.871773 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82nb\" (UniqueName: \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.871876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.871824 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.872688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.872663 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.875152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.875128 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.937006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.935638 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82nb\" (UniqueName: \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\") pod \"maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:18.960002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:18.959929 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:19.176697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.176601 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:19.176697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.176675 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:19.180285 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.180252 4486 scope.go:122] "RemoveContainer" containerID="612ef46ba81c15f3f137e3b0e39b2e7ad8c5120a1b2586e644d04988b7e0f1d6" Nov 04 01:28:19.202657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.202638 4486 scope.go:122] "RemoveContainer" containerID="d8eba2e1aa08a43f2a1804387f48ec320041e8bb6e304d6d698a49ba487eeff9" Nov 04 01:28:19.213811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.213786 4486 scope.go:122] "RemoveContainer" containerID="10d2e7bca2ed97a700e52a1ada66d05d3196ad8cfe970ecc1e0480abd3e748a2" Nov 04 01:28:19.224509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.224491 4486 scope.go:122] "RemoveContainer" containerID="549fa0a8f2174821e3b6c044b21c824e6c36e4b542e9099c61333ab1b9937376" Nov 04 01:28:19.270970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.270941 4486 scope.go:122] "RemoveContainer" containerID="c235b9f95fa8d4350c339f4a7797a5dd9d6ff0beab605fe0669f4f8c9c0c9828" Nov 04 01:28:19.284500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.284479 4486 scope.go:122] "RemoveContainer" containerID="7934ad14399e67e30a02a6b99f87fab87cf1aa52c6618b0d9c2ff43ad4abc966" Nov 04 01:28:19.294860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.294837 4486 scope.go:122] "RemoveContainer" containerID="34d4c7550d2ee34fdc9e9127a479bab59540f8b0976db29a4c8927f8b21cd85c" Nov 04 01:28:19.315783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.315759 4486 scope.go:122] "RemoveContainer" containerID="fbd826342e9e9127d9efd8c2f418fee05ab25178d1eb164d251a13092846fd98" Nov 04 01:28:19.365467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.365431 4486 scope.go:122] "RemoveContainer" containerID="6df1ebfcd25629642a79bd99a54c7b0f26f7af7a081ad466623db16de56a7e6c" Nov 04 01:28:19.381679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.381647 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl"] Nov 04 01:28:19.383105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.383086 4486 scope.go:122] "RemoveContainer" containerID="155babbce3f04f1ba6a604c4fd9e633cdf1d90960fa686199e185fc1aae36af7" Nov 04 01:28:19.408173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.408152 4486 scope.go:122] "RemoveContainer" containerID="cc5a2740c3b764b5fa989e1cb090e9624678c53e84eefd23ff3f79caf8ede985" Nov 04 01:28:19.420359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.420338 4486 scope.go:122] "RemoveContainer" containerID="fe6580a4b67fd492dd45b5696b33b5e49ac8cca8a828fa02c4bbd3ffe865465d" Nov 04 01:28:19.470261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.470235 4486 scope.go:122] "RemoveContainer" containerID="0020c36edc67ae9ead94f56560bef9a21c1f7fc8016315564a858ecfabf6b109" Nov 04 01:28:19.482067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.481976 4486 scope.go:122] "RemoveContainer" containerID="d2af75582248023decc03db37f577074636440453c43ecc7d25cbfb2c701d56b" Nov 04 01:28:19.492257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.492238 4486 scope.go:122] "RemoveContainer" containerID="66e84e4ac71d12f7e50c61a2ce9471c99780db9b52c21b4960fcf9a7ec8f54d6" Nov 04 01:28:19.504263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.504242 4486 scope.go:122] "RemoveContainer" containerID="68bba6078465c4dd9b93d1c6b9272cd924c91acbbfce61420e6c12217d6d796e" Nov 04 01:28:19.993031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.992898 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" event={"ID":"d49648e5-fc2e-4c81-96d8-3c542b04ad81","Type":"ContainerStarted","Data":"613abca881219e7b58f73703e57e634b3031ef0aa167beed008086046a18193a"} Nov 04 01:28:19.993031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:19.992992 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" event={"ID":"d49648e5-fc2e-4c81-96d8-3c542b04ad81","Type":"ContainerStarted","Data":"b0bc8f3a321dbe2952b95b14b0aa9f0bd429469bbef30b9294c6e5892624558c"} Nov 04 01:28:20.112400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:20.112342 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" podStartSLOduration=2.112327964 podStartE2EDuration="2.112327964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:20.109853809 +0000 UTC m=+1868.868515971" watchObservedRunningTime="2026-11-04 01:28:20.112327964 +0000 UTC m=+1868.870990125" Nov 04 01:28:20.177061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:20.177019 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:20.177235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:20.177082 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:21.003684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.003601 4486 generic.go:356] "Generic (PLEG): container finished" podID="d49648e5-fc2e-4c81-96d8-3c542b04ad81" containerID="613abca881219e7b58f73703e57e634b3031ef0aa167beed008086046a18193a" exitCode=0 Nov 04 01:28:21.004086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.003687 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" event={"ID":"d49648e5-fc2e-4c81-96d8-3c542b04ad81","Type":"ContainerDied","Data":"613abca881219e7b58f73703e57e634b3031ef0aa167beed008086046a18193a"} Nov 04 01:28:21.104837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.104799 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" containerID="cri-o://11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61" gracePeriod=45 Nov 04 01:28:21.105030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.104799 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" containerID="cri-o://21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f" gracePeriod=45 Nov 04 01:28:21.179781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.176872 4486 patch_prober.go:29] interesting pod/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Nov 04 01:28:21.179781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.176939 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Nov 04 01:28:21.583646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.583576 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:28:21.585768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.585728 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:28:21.721076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721043 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\" (UniqueName: \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.721243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721102 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721212 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.721417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721288 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\" (UniqueName: \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721357 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721368 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.721583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721420 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721472 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.721583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721524 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.721762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721602 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721664 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.721762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721689 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\") pod \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\" (UID: \"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098\") " Nov 04 01:28:21.721932 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.721773 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\") pod \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\" (UID: \"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e\") " Nov 04 01:28:21.722797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.722397 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.722797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.722709 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.722797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.722731 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.723011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.722919 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.723204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.723109 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.723204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.723166 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.723473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.723287 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.724274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.724240 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "kube-api-access-2bj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:21.725664 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.725635 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:21.725757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.725696 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "kube-api-access-cjtnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:21.725951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.725915 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm" pod "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" (UID: "f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.726241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.726213 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:21.726432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.726413 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm" pod "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" (UID: "c83c095d-a66c-4f10-9f6c-cb3d6c2ff098"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:21.825145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825104 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825148 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bj5j\" (UniqueName: \"kubernetes.io/projected/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-kube-api-access-2bj5j\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825164 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825177 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825190 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825203 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825216 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825230 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825243 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:21.825409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:21.825256 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjtnh\" (UniqueName: \"kubernetes.io/projected/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e-kube-api-access-cjtnh\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:22.015957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.015862 4486 generic.go:356] "Generic (PLEG): container finished" podID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerID="21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f" exitCode=0 Nov 04 01:28:22.015957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.015909 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" Nov 04 01:28:22.016494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.015943 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" event={"ID":"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e","Type":"ContainerDied","Data":"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f"} Nov 04 01:28:22.016494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.016043 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" event={"ID":"f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e","Type":"ContainerDied","Data":"2bc2689eaf616242420300a0c2dda1c537ea056f448f7e9ec42af260678b7c3e"} Nov 04 01:28:22.016494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.016079 4486 scope.go:122] "RemoveContainer" containerID="21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f" Nov 04 01:28:22.018059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.018025 4486 generic.go:356] "Generic (PLEG): container finished" podID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerID="11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61" exitCode=0 Nov 04 01:28:22.018224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.018069 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" Nov 04 01:28:22.018483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.018437 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" event={"ID":"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098","Type":"ContainerDied","Data":"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61"} Nov 04 01:28:22.019308 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.018481 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6" event={"ID":"c83c095d-a66c-4f10-9f6c-cb3d6c2ff098","Type":"ContainerDied","Data":"e944ab9565b1ac00547bc990dc083da5f726e4f59436fcd48cdf7b0cab75de5c"} Nov 04 01:28:22.043125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.043106 4486 scope.go:122] "RemoveContainer" containerID="21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f" Nov 04 01:28:22.043401 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:22.043379 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f\": container with ID starting with 21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f not found: ID does not exist" containerID="21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f" Nov 04 01:28:22.043486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.043409 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f"} err="failed to get container status \"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f\": rpc error: code = NotFound desc = could not find container \"21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f\": container with ID starting with 21c88a891526deb0682f67050266ae26b9e5dfc89bc619733461d59cf5701f7f not found: ID does not exist" Nov 04 01:28:22.043486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.043428 4486 scope.go:122] "RemoveContainer" containerID="11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61" Nov 04 01:28:22.057516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.057493 4486 scope.go:122] "RemoveContainer" containerID="11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61" Nov 04 01:28:22.057850 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:22.057822 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61\": container with ID starting with 11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61 not found: ID does not exist" containerID="11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61" Nov 04 01:28:22.057943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.057862 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61"} err="failed to get container status \"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61\": rpc error: code = NotFound desc = could not find container \"11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61\": container with ID starting with 11eae7d6f474957a82d1d7acef0bc5b4ed73cf1359316df14462e6bbb9eede61 not found: ID does not exist" Nov 04 01:28:22.186371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.186256 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:28:22.190659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.190631 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:28:22.199001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.191788 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:28:22.204921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.204593 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:28:22.205870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.205843 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj"] Nov 04 01:28:22.255878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.255841 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:22.313147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.313112 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:28:22.323317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.323281 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c563ee/distinct-2-w1-c563ee-kserve-69fdfc7f94-bbfh6"] Nov 04 01:28:22.334883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.334848 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\" (UniqueName: \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\") pod \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " Nov 04 01:28:22.335057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.334964 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\") pod \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " Nov 04 01:28:22.335120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.335068 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\") pod \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\" (UID: \"d49648e5-fc2e-4c81-96d8-3c542b04ad81\") " Nov 04 01:28:22.335647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.335616 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca" pod "d49648e5-fc2e-4c81-96d8-3c542b04ad81" (UID: "d49648e5-fc2e-4c81-96d8-3c542b04ad81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:22.338394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.338363 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:22.350585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.350558 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb" pod "d49648e5-fc2e-4c81-96d8-3c542b04ad81" (UID: "d49648e5-fc2e-4c81-96d8-3c542b04ad81"). InnerVolumeSpecName "kube-api-access-c82nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:22.351621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.350689 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca" pod "d49648e5-fc2e-4c81-96d8-3c542b04ad81" (UID: "d49648e5-fc2e-4c81-96d8-3c542b04ad81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:22.471730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.456245 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49648e5-fc2e-4c81-96d8-3c542b04ad81-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:22.471730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.471505 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82nb\" (UniqueName: \"kubernetes.io/projected/d49648e5-fc2e-4c81-96d8-3c542b04ad81-kube-api-access-c82nb\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:22.540857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.539896 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c563ee/distinct-w1-c563ee-kserve-5d696b7c49-2jrgj" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": context deadline exceeded" Nov 04 01:28:22.627463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.627373 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:28:22.660003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.657321 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" containerID="cri-o://55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b" gracePeriod=30 Nov 04 01:28:22.660003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.658532 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:28:22.660003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.658721 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" containerID="cri-o://a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa" gracePeriod=30 Nov 04 01:28:22.669233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.665992 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:28:22.669233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.666186 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" containerID="cri-o://1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406" gracePeriod=30 Nov 04 01:28:22.675916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.675884 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:28:22.679194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:22.679145 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" containerID="cri-o://36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb" gracePeriod=30 Nov 04 01:28:23.043242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:23.043155 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" Nov 04 01:28:23.043242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:23.043187 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl" event={"ID":"d49648e5-fc2e-4c81-96d8-3c542b04ad81","Type":"ContainerDied","Data":"b0bc8f3a321dbe2952b95b14b0aa9f0bd429469bbef30b9294c6e5892624558c"} Nov 04 01:28:23.043758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:23.043247 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0bc8f3a321dbe2952b95b14b0aa9f0bd429469bbef30b9294c6e5892624558c" Nov 04 01:28:23.388360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:23.388322 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" path="/var/lib/kubelet/pods/c83c095d-a66c-4f10-9f6c-cb3d6c2ff098/volumes" Nov 04 01:28:23.389537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:23.389455 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" path="/var/lib/kubelet/pods/f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e/volumes" Nov 04 01:28:25.267010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.265957 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:25.267010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.266035 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:25.921848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.921817 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:28:25.922749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.922730 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d49648e5-fc2e-4c81-96d8-3c542b04ad81" containerName="revoke-keys" Nov 04 01:28:25.922870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.922846 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49648e5-fc2e-4c81-96d8-3c542b04ad81" containerName="revoke-keys" Nov 04 01:28:25.922955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.922946 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" Nov 04 01:28:25.923028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923019 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" Nov 04 01:28:25.923124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923116 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" Nov 04 01:28:25.923176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923170 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" Nov 04 01:28:25.923380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923371 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83c095d-a66c-4f10-9f6c-cb3d6c2ff098" containerName="main" Nov 04 01:28:25.923441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923434 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ec57e6-6f7b-4cb0-80a4-a3cb0263d61e" containerName="main" Nov 04 01:28:25.923496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.923489 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49648e5-fc2e-4c81-96d8-3c542b04ad81" containerName="revoke-keys" Nov 04 01:28:25.938006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:25.929115 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.033009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.032561 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-42958c-openshift-default-dockercfg-99cxr\"" Nov 04 01:28:26.062006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.061698 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206676 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z94q\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206727 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206761 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206819 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206886 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206913 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.206950 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.207080 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.207714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.207131 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314481 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314546 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314621 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314708 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314739 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314782 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.314932 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.315014 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.323092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.315223 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z94q\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.353045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.336235 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.507127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.452502 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.587040 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.558851 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.592808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.592766 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.607174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.606816 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.607174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.607121 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.607726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.607688 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.670786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.669740 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z94q\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.706344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.706308 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\") pod \"e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:26.755084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.755044 4486 patch_prober.go:29] interesting pod/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:26.755317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.755291 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:26.993006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:26.992530 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:27.503534 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:27.503499 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf228b269_e7f1_41f0_9c3f_76e24351c64d.slice/crio-dac0ea2d74499617e14f2759d4b7caa8050e87e5d03dc31a5b192f0f3f0e6275 WatchSource:0}: Error finding container dac0ea2d74499617e14f2759d4b7caa8050e87e5d03dc31a5b192f0f3f0e6275: Status 404 returned error can't find the container with id dac0ea2d74499617e14f2759d4b7caa8050e87e5d03dc31a5b192f0f3f0e6275 Nov 04 01:28:27.503943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:27.503910 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:28:27.506330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:27.506294 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:27.506434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:27.506367 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:27.506434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:27.506401 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:28.025199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.025175 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124010 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124072 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124108 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124196 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124276 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124372 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124405 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124582 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.124694 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\") pod \"92966232-2a3d-4a7e-896c-c77096e8fdd1\" (UID: \"92966232-2a3d-4a7e-896c-c77096e8fdd1\") " Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.126767 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.129182 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.129692 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.129906 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.130247 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.131448 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.131474 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.131489 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.131504 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.136007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.131518 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92966232-2a3d-4a7e-896c-c77096e8fdd1-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.175011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.158258 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.175011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.158394 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "kube-api-access-xrb7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.175011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.164387 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:28.175011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.164835 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token" pod "92966232-2a3d-4a7e-896c-c77096e8fdd1" (UID: "92966232-2a3d-4a7e-896c-c77096e8fdd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.233294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.233259 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.233294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.233295 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrb7c\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-kube-api-access-xrb7c\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.233525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.233311 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.233525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.233327 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92966232-2a3d-4a7e-896c-c77096e8fdd1-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.255133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.255103 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:28:28.257619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.257595 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:28:28.262770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.262751 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:28:28.333797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.333763 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.333832 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.333856 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.334023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.333914 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.334023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.333940 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334057 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334126 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334168 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334200 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334219 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334263 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334277 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334424 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334452 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334474 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334501 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334512 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334560 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334588 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334617 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.334662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334653 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334705 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334778 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334821 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334835 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334871 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334902 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\") pod \"94101ccd-0f20-4e5c-8226-7adb50643c66\" (UID: \"94101ccd-0f20-4e5c-8226-7adb50643c66\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334911 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.334957 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335064 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335100 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335144 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335188 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335224 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335266 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\") pod \"0c58f026-1d30-4188-9bb0-251c1a968b20\" (UID: \"0c58f026-1d30-4188-9bb0-251c1a968b20\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335264 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335278 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335296 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\") pod \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\" (UID: \"d93c59d3-186c-4e77-a8ae-40c1e4102cc4\") " Nov 04 01:28:28.335558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335471 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.335741 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336255 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336279 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336293 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336308 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336323 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336340 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c58f026-1d30-4188-9bb0-251c1a968b20-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336354 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94101ccd-0f20-4e5c-8226-7adb50643c66-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336370 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336384 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336400 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336413 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336427 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.336688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336440 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.337459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336911 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.337459 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.336966 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.337817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.337780 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:28.338228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.338188 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "kube-api-access-sp44f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.338565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.338528 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:28.338896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.338803 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "kube-api-access-gztpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.339355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339293 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.339355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339330 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:28.339493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339414 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.339573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339547 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.339624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339586 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.339811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.339788 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token" pod "94101ccd-0f20-4e5c-8226-7adb50643c66" (UID: "94101ccd-0f20-4e5c-8226-7adb50643c66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.340133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.340115 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v" pod "0c58f026-1d30-4188-9bb0-251c1a968b20" (UID: "0c58f026-1d30-4188-9bb0-251c1a968b20"). InnerVolumeSpecName "kube-api-access-h6b9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:28.340655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.340641 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy" pod "d93c59d3-186c-4e77-a8ae-40c1e4102cc4" (UID: "d93c59d3-186c-4e77-a8ae-40c1e4102cc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441085 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441115 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441131 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441147 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441178 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441193 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441207 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441222 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441236 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441250 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94101ccd-0f20-4e5c-8226-7adb50643c66-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441263 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c58f026-1d30-4188-9bb0-251c1a968b20-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441277 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp44f\" (UniqueName: \"kubernetes.io/projected/d93c59d3-186c-4e77-a8ae-40c1e4102cc4-kube-api-access-sp44f\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441290 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztpx\" (UniqueName: \"kubernetes.io/projected/94101ccd-0f20-4e5c-8226-7adb50643c66-kube-api-access-gztpx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.442458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.441304 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b9v\" (UniqueName: \"kubernetes.io/projected/0c58f026-1d30-4188-9bb0-251c1a968b20-kube-api-access-h6b9v\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:28.574405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.574364 4486 generic.go:356] "Generic (PLEG): container finished" podID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerID="a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa" exitCode=0 Nov 04 01:28:28.574887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.574513 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerDied","Data":"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa"} Nov 04 01:28:28.574887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.574617 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" event={"ID":"92966232-2a3d-4a7e-896c-c77096e8fdd1","Type":"ContainerDied","Data":"ab28cc5cd2967396f4011d2718219bb0f2e8fcd0953b583a0df7ea5b53ccf0ab"} Nov 04 01:28:28.574887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.574659 4486 scope.go:122] "RemoveContainer" containerID="a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa" Nov 04 01:28:28.574887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.574878 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh" Nov 04 01:28:28.593685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.593193 4486 scope.go:122] "RemoveContainer" containerID="4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb" Nov 04 01:28:28.593685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.593520 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerStarted","Data":"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925"} Nov 04 01:28:28.593685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.593562 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerStarted","Data":"dac0ea2d74499617e14f2759d4b7caa8050e87e5d03dc31a5b192f0f3f0e6275"} Nov 04 01:28:28.595877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.595850 4486 generic.go:356] "Generic (PLEG): container finished" podID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerID="36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb" exitCode=0 Nov 04 01:28:28.596065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.595909 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" Nov 04 01:28:28.596065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.595952 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" event={"ID":"d93c59d3-186c-4e77-a8ae-40c1e4102cc4","Type":"ContainerDied","Data":"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb"} Nov 04 01:28:28.596065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.596004 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6" event={"ID":"d93c59d3-186c-4e77-a8ae-40c1e4102cc4","Type":"ContainerDied","Data":"3ac1fa4065552f5a1075cd293ee628017a62fdb7faaaac2dd4737749a1d447e0"} Nov 04 01:28:28.597883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.597863 4486 generic.go:356] "Generic (PLEG): container finished" podID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerID="1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406" exitCode=0 Nov 04 01:28:28.599029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.598151 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" Nov 04 01:28:28.599171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.599102 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" event={"ID":"0c58f026-1d30-4188-9bb0-251c1a968b20","Type":"ContainerDied","Data":"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406"} Nov 04 01:28:28.599171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.599158 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2" event={"ID":"0c58f026-1d30-4188-9bb0-251c1a968b20","Type":"ContainerDied","Data":"85e773ddbfae0d78df712d2477d81d2d887741a5d762eae52b8cee65d068fa03"} Nov 04 01:28:28.604687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.604666 4486 generic.go:356] "Generic (PLEG): container finished" podID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerID="55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b" exitCode=0 Nov 04 01:28:28.604899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.604884 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" event={"ID":"94101ccd-0f20-4e5c-8226-7adb50643c66","Type":"ContainerDied","Data":"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b"} Nov 04 01:28:28.605252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.605232 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" event={"ID":"94101ccd-0f20-4e5c-8226-7adb50643c66","Type":"ContainerDied","Data":"8d15608f4af2d2abb236ee8daba17dc2fa94f1b628915ce40ea83f521104fc51"} Nov 04 01:28:28.605851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.605835 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k" Nov 04 01:28:28.642200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.642132 4486 scope.go:122] "RemoveContainer" containerID="a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa" Nov 04 01:28:28.642503 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:28.642475 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa\": container with ID starting with a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa not found: ID does not exist" containerID="a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa" Nov 04 01:28:28.642674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.642507 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa"} err="failed to get container status \"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa\": rpc error: code = NotFound desc = could not find container \"a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa\": container with ID starting with a4a8f6700258161f6c4260d7132bc75522389869998adfcf369449dcefd5acaa not found: ID does not exist" Nov 04 01:28:28.642674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.642526 4486 scope.go:122] "RemoveContainer" containerID="4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb" Nov 04 01:28:28.642884 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:28.642863 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb\": container with ID starting with 4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb not found: ID does not exist" containerID="4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb" Nov 04 01:28:28.642967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.642888 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb"} err="failed to get container status \"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb\": rpc error: code = NotFound desc = could not find container \"4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb\": container with ID starting with 4d668a9b5316cb829d646859c6a7e7aae5bf5e4b4d9cc813afa3580e0daa6cbb not found: ID does not exist" Nov 04 01:28:28.642967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.642902 4486 scope.go:122] "RemoveContainer" containerID="36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb" Nov 04 01:28:28.680260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.680224 4486 scope.go:122] "RemoveContainer" containerID="36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb" Nov 04 01:28:28.680654 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:28.680632 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb\": container with ID starting with 36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb not found: ID does not exist" containerID="36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb" Nov 04 01:28:28.680768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.680663 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb"} err="failed to get container status \"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb\": rpc error: code = NotFound desc = could not find container \"36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb\": container with ID starting with 36810aabd93c64a03ce1d9b9ef5324a9a4e01e6f304d7b0105d350e7ae9fcbdb not found: ID does not exist" Nov 04 01:28:28.680768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.680679 4486 scope.go:122] "RemoveContainer" containerID="1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406" Nov 04 01:28:28.695155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.695131 4486 scope.go:122] "RemoveContainer" containerID="1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406" Nov 04 01:28:28.695461 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:28.695434 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406\": container with ID starting with 1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406 not found: ID does not exist" containerID="1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406" Nov 04 01:28:28.695578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.695470 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406"} err="failed to get container status \"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406\": rpc error: code = NotFound desc = could not find container \"1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406\": container with ID starting with 1134d3cf171012a46148cbe9a93d027123652c8c6c8e776934a8e9fe2c9f0406 not found: ID does not exist" Nov 04 01:28:28.695578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.695494 4486 scope.go:122] "RemoveContainer" containerID="55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b" Nov 04 01:28:28.706581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.706562 4486 scope.go:122] "RemoveContainer" containerID="55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b" Nov 04 01:28:28.706854 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:28.706832 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b\": container with ID starting with 55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b not found: ID does not exist" containerID="55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b" Nov 04 01:28:28.706927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.706867 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b"} err="failed to get container status \"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b\": rpc error: code = NotFound desc = could not find container \"55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b\": container with ID starting with 55b7dd9cfccd41f2aa38681d4c91282eb42978098fe3e191805b474a2ce8b55b not found: ID does not exist" Nov 04 01:28:28.795375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.795326 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podStartSLOduration=3.79531315 podStartE2EDuration="3.79531315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:28.792930486 +0000 UTC m=+1877.551592649" watchObservedRunningTime="2026-11-04 01:28:28.79531315 +0000 UTC m=+1877.553975308" Nov 04 01:28:28.905908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.905831 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:28:28.931161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.931130 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-mgns6"] Nov 04 01:28:28.993303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.993274 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:28.998690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:28.998661 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:29.205564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.205454 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:28:29.210547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.210513 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-4rrfh"] Nov 04 01:28:29.236916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.236423 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:28:29.244005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.243954 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-kjgf2"] Nov 04 01:28:29.351855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.351824 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" path="/var/lib/kubelet/pods/0c58f026-1d30-4188-9bb0-251c1a968b20/volumes" Nov 04 01:28:29.352591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.352566 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" path="/var/lib/kubelet/pods/92966232-2a3d-4a7e-896c-c77096e8fdd1/volumes" Nov 04 01:28:29.353145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.353129 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" path="/var/lib/kubelet/pods/d93c59d3-186c-4e77-a8ae-40c1e4102cc4/volumes" Nov 04 01:28:29.459194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.459094 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:28:29.464321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.464290 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2fd27d-openshift-default-77cb48fb59-44v9k"] Nov 04 01:28:29.618514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.618483 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:29.619626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:29.619607 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:28:30.165886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:30.165845 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7"] Nov 04 01:28:30.170358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:30.170329 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2ee045-9d2d7"] Nov 04 01:28:31.436009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:31.435797 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" path="/var/lib/kubelet/pods/94101ccd-0f20-4e5c-8226-7adb50643c66/volumes" Nov 04 01:28:31.440029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:31.437039 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8883bf0-c41c-45c6-8e24-82e2ced9aca4" path="/var/lib/kubelet/pods/e8883bf0-c41c-45c6-8e24-82e2ced9aca4/volumes" Nov 04 01:28:37.194009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.193783 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:28:37.194441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.194058 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" containerID="cri-o://ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a" gracePeriod=30 Nov 04 01:28:37.223757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.223710 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:28:37.224018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.223975 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" containerID="cri-o://23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca" gracePeriod=30 Nov 04 01:28:37.228966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.228933 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:28:37.229242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.229191 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" containerID="cri-o://350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded" gracePeriod=30 Nov 04 01:28:37.273365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.273327 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:28:37.273632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:37.273603 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" containerID="cri-o://1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124" gracePeriod=30 Nov 04 01:28:38.747960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:38.747917 4486 patch_prober.go:29] interesting pod/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:38.748383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:38.748032 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:39.004883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.004796 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:28:39.005104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.005030 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" containerID="cri-o://3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a" gracePeriod=30 Nov 04 01:28:39.036850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.036813 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:28:39.037087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.037061 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" containerID="cri-o://95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d" gracePeriod=30 Nov 04 01:28:39.041650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.041624 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:28:39.041810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:39.041792 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" containerID="cri-o://c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b" gracePeriod=30 Nov 04 01:28:41.717456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:41.717413 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:41.717837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:41.717470 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:42.590478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.590181 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591175 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591349 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591433 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591474 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591506 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.591583 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.592803 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.593137 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.593551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.593427 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.595246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.595214 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.602348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.602320 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:42.602794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.602774 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.642937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.642905 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:28:42.646974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.646948 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:28:42.652750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.652726 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:28:42.693536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693509 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.693646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693545 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.693646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693585 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.693646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693618 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.693824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693661 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.693824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693700 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.693824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693730 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.693824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693758 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.693824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693796 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693822 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693837 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693851 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.693932 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694009 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694025 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694014 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694060 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.694134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694108 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694169 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\") pod \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\" (UID: \"b106dfc4-ef90-4c74-a481-91c119cf4c4c\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694217 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694254 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694289 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694321 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694338 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694361 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694385 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694422 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694471 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694499 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694534 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.694587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694583 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694612 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\") pod \"545a415b-07e6-425b-a489-f269781812f2\" (UID: \"545a415b-07e6-425b-a489-f269781812f2\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694614 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694696 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694730 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694809 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\") pod \"07be9abf-2036-4221-805a-b1b856ac1be3\" (UID: \"07be9abf-2036-4221-805a-b1b856ac1be3\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694815 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694888 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.694940 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.695345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695283 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.695907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695881 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695914 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695929 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695944 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695961 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.695976 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696006 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696017 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be9abf-2036-4221-805a-b1b856ac1be3-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696033 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696046 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696066 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696081 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696095 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545a415b-07e6-425b-a489-f269781812f2-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696112 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696128 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696145 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696160 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696414 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.696606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696475 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.696630 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "kube-api-access-twxfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.697105 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.697176 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.697864 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.698242 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:42.699557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.699268 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.699888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.699855 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:42.700037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.699859 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq" pod "b106dfc4-ef90-4c74-a481-91c119cf4c4c" (UID: "b106dfc4-ef90-4c74-a481-91c119cf4c4c"). InnerVolumeSpecName "kube-api-access-vbvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.700191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700077 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.700191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700163 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.700351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700192 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.700443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700412 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:42.700921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700886 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy" pod "07be9abf-2036-4221-805a-b1b856ac1be3" (UID: "07be9abf-2036-4221-805a-b1b856ac1be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.700921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.700896 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "kube-api-access-zlbw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.701266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.701235 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.701506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.701486 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7" pod "545a415b-07e6-425b-a489-f269781812f2" (UID: "545a415b-07e6-425b-a489-f269781812f2"). InnerVolumeSpecName "kube-api-access-8qfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:42.795248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.795161 4486 generic.go:356] "Generic (PLEG): container finished" podID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerID="350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded" exitCode=0 Nov 04 01:28:42.795248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.795224 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" Nov 04 01:28:42.795781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.795268 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" event={"ID":"f3891318-4bb2-4c56-bea8-f6a314fe7fcb","Type":"ContainerDied","Data":"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded"} Nov 04 01:28:42.795781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.795331 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr" event={"ID":"f3891318-4bb2-4c56-bea8-f6a314fe7fcb","Type":"ContainerDied","Data":"f03a157fa06b8f938ec771188872d730cd2a979758d199774dbbc78f620e0201"} Nov 04 01:28:42.795781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.795365 4486 scope.go:122] "RemoveContainer" containerID="350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded" Nov 04 01:28:42.796648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.796630 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\") pod \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\" (UID: \"f3891318-4bb2-4c56-bea8-f6a314fe7fcb\") " Nov 04 01:28:42.796886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.796863 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket" pod "f3891318-4bb2-4c56-bea8-f6a314fe7fcb" (UID: "f3891318-4bb2-4c56-bea8-f6a314fe7fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:42.797048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797028 4486 generic.go:356] "Generic (PLEG): container finished" podID="07be9abf-2036-4221-805a-b1b856ac1be3" containerID="1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124" exitCode=0 Nov 04 01:28:42.797162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797143 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" Nov 04 01:28:42.797218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797169 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" event={"ID":"07be9abf-2036-4221-805a-b1b856ac1be3","Type":"ContainerDied","Data":"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124"} Nov 04 01:28:42.797218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797204 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh" event={"ID":"07be9abf-2036-4221-805a-b1b856ac1be3","Type":"ContainerDied","Data":"f4686cc3b08128f34734bb525e3004bb4103e29f6d12d3abe00e61d100db29bb"} Nov 04 01:28:42.797526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797509 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797532 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qfr7\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-kube-api-access-8qfr7\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797551 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbw8\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-kube-api-access-zlbw8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797560 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797572 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797585 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797600 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797613 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be9abf-2036-4221-805a-b1b856ac1be3-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797627 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545a415b-07e6-425b-a489-f269781812f2-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797641 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797653 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxfr\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-kube-api-access-twxfr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797667 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbvmq\" (UniqueName: \"kubernetes.io/projected/b106dfc4-ef90-4c74-a481-91c119cf4c4c-kube-api-access-vbvmq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797681 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be9abf-2036-4221-805a-b1b856ac1be3-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797694 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b106dfc4-ef90-4c74-a481-91c119cf4c4c-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797702 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797716 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545a415b-07e6-425b-a489-f269781812f2-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797727 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3891318-4bb2-4c56-bea8-f6a314fe7fcb-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797735 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be9abf-2036-4221-805a-b1b856ac1be3-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.797807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.797744 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545a415b-07e6-425b-a489-f269781812f2-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:42.798955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.798930 4486 generic.go:356] "Generic (PLEG): container finished" podID="545a415b-07e6-425b-a489-f269781812f2" containerID="23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca" exitCode=0 Nov 04 01:28:42.799087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.798965 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" Nov 04 01:28:42.799157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.799093 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" event={"ID":"545a415b-07e6-425b-a489-f269781812f2","Type":"ContainerDied","Data":"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca"} Nov 04 01:28:42.799157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.799149 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz" event={"ID":"545a415b-07e6-425b-a489-f269781812f2","Type":"ContainerDied","Data":"cd6e5d80187946844ef962a158cdf34873b7b5dce731915ff0e9a1394e340265"} Nov 04 01:28:42.803197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.803173 4486 generic.go:356] "Generic (PLEG): container finished" podID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerID="ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a" exitCode=0 Nov 04 01:28:42.803319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.803209 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" Nov 04 01:28:42.803319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.803220 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" event={"ID":"b106dfc4-ef90-4c74-a481-91c119cf4c4c","Type":"ContainerDied","Data":"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a"} Nov 04 01:28:42.803319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.803256 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh" event={"ID":"b106dfc4-ef90-4c74-a481-91c119cf4c4c","Type":"ContainerDied","Data":"2b72cdedfe1f4955e5e1cf6d65fe6ecc5719972bd8e9169c03dadd3bf10b17c7"} Nov 04 01:28:42.810919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.809820 4486 scope.go:122] "RemoveContainer" containerID="350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded" Nov 04 01:28:42.810919 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:42.810156 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded\": container with ID starting with 350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded not found: ID does not exist" containerID="350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded" Nov 04 01:28:42.810919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.810182 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded"} err="failed to get container status \"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded\": rpc error: code = NotFound desc = could not find container \"350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded\": container with ID starting with 350935603fba300dafade7041c0f0ec160fc1b3162758af118a4487adad22ded not found: ID does not exist" Nov 04 01:28:42.810919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.810200 4486 scope.go:122] "RemoveContainer" containerID="1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124" Nov 04 01:28:42.824685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.824660 4486 scope.go:122] "RemoveContainer" containerID="1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124" Nov 04 01:28:42.825039 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:42.825015 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124\": container with ID starting with 1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124 not found: ID does not exist" containerID="1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124" Nov 04 01:28:42.825145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.825049 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124"} err="failed to get container status \"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124\": rpc error: code = NotFound desc = could not find container \"1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124\": container with ID starting with 1c12377c42f3ab6a409cac469cbaf79db5432c61f4319696accb4fa9e3f3b124 not found: ID does not exist" Nov 04 01:28:42.825145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.825075 4486 scope.go:122] "RemoveContainer" containerID="23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca" Nov 04 01:28:42.838102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.837689 4486 scope.go:122] "RemoveContainer" containerID="23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca" Nov 04 01:28:42.838102 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:42.837992 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca\": container with ID starting with 23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca not found: ID does not exist" containerID="23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca" Nov 04 01:28:42.838102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.838023 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca"} err="failed to get container status \"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca\": rpc error: code = NotFound desc = could not find container \"23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca\": container with ID starting with 23102cc0626fca17c06f0e5b3746c89f67bf79a0942ddfb6f4006c6890e147ca not found: ID does not exist" Nov 04 01:28:42.838102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.838038 4486 scope.go:122] "RemoveContainer" containerID="ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a" Nov 04 01:28:42.848442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.848425 4486 scope.go:122] "RemoveContainer" containerID="ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a" Nov 04 01:28:42.848693 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:42.848673 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a\": container with ID starting with ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a not found: ID does not exist" containerID="ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a" Nov 04 01:28:42.848751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:42.848701 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a"} err="failed to get container status \"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a\": rpc error: code = NotFound desc = could not find container \"ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a\": container with ID starting with ab830cd97ca47956d3e0755a4309c8d63f585deb0cd155a63c5f0cf59a25a09a not found: ID does not exist" Nov 04 01:28:43.296492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.296441 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:28:43.301492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.301460 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-tqrvh"] Nov 04 01:28:43.361362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.361284 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" path="/var/lib/kubelet/pods/b106dfc4-ef90-4c74-a481-91c119cf4c4c/volumes" Nov 04 01:28:43.400296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.400256 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:28:43.408513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.408486 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-j6wvh"] Nov 04 01:28:43.513728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.513693 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:28:43.518578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.518543 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-frnhz"] Nov 04 01:28:43.596190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.596151 4486 patch_prober.go:29] interesting pod/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:28:43.596361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.596217 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:28:43.627208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.627129 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:28:43.633970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:43.633947 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-678aaf-openshift-default-5b9b5966c5-pm4gr"] Nov 04 01:28:43.833939 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:43.833901 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:28:44.411904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.411863 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.424132 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.424541 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.424616 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.425249 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.425572 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.425877 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.425937 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.426304 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.426643 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.426748 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.426783 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.427103 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.430608 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.430858 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\") pod \"af563074-bd2d-4b27-a57d-0a1f9d26b186\" (UID: \"af563074-bd2d-4b27-a57d-0a1f9d26b186\") " Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.431713 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.431733 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.431747 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.431763 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.431776 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af563074-bd2d-4b27-a57d-0a1f9d26b186-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.437353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.432145 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.438632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.438474 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.438764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.438667 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.440135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.440065 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd" pod "af563074-bd2d-4b27-a57d-0a1f9d26b186" (UID: "af563074-bd2d-4b27-a57d-0a1f9d26b186"). InnerVolumeSpecName "kube-api-access-9j4nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.462244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.462217 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:28:44.512831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.512804 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:28:44.533149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533127 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.533252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533161 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533195 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.533338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533290 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533337 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533380 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.533498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533409 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.533498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533452 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.533498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533488 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533540 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.533661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533590 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533619 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533712 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.533823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533751 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.533823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533780 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.533823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533801 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533833 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533866 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533883 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\") pod \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\" (UID: \"1c271a6d-5a7a-4ca5-a273-845c84488ef5\") " Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533920 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533973 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.534039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.533994 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.534366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534043 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.534366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534084 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\") pod \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\" (UID: \"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c\") " Nov 04 01:28:44.534366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534186 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.534521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534476 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.534739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534721 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.534808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534745 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.534808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534761 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j4nd\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-kube-api-access-9j4nd\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.534808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534777 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.534808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534791 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534807 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534821 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534837 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534852 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534867 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534882 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.534896 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af563074-bd2d-4b27-a57d-0a1f9d26b186-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.535609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.535579 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.535709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.535643 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.536210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536180 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.536342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536237 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.536342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536316 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.536438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536376 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:28:44.536438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536416 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.536838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.536824 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "kube-api-access-29pb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.537157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.537143 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy" pod "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" (UID: "7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:28:44.537683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.537665 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft" pod "1c271a6d-5a7a-4ca5-a273-845c84488ef5" (UID: "1c271a6d-5a7a-4ca5-a273-845c84488ef5"). InnerVolumeSpecName "kube-api-access-hqbft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:44.637169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637086 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637357 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637465 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637554 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29pb6\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-kube-api-access-29pb6\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637678 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637665 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637753 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637838 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.637934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.637923 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.638037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.638023 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.638140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.638127 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqbft\" (UniqueName: \"kubernetes.io/projected/1c271a6d-5a7a-4ca5-a273-845c84488ef5-kube-api-access-hqbft\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:44.857768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.857737 4486 generic.go:356] "Generic (PLEG): container finished" podID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerID="c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b" exitCode=0 Nov 04 01:28:44.858246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.857787 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" Nov 04 01:28:44.858246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.857830 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" event={"ID":"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c","Type":"ContainerDied","Data":"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b"} Nov 04 01:28:44.858246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.857885 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s" event={"ID":"7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c","Type":"ContainerDied","Data":"851939da9128c3168dae0083eb63cb2cce087465ba1d5c1c47b3e6cbdcd024e7"} Nov 04 01:28:44.858246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.857916 4486 scope.go:122] "RemoveContainer" containerID="c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b" Nov 04 01:28:44.859796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.859772 4486 generic.go:356] "Generic (PLEG): container finished" podID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerID="95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d" exitCode=0 Nov 04 01:28:44.859915 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.859823 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" Nov 04 01:28:44.859915 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.859861 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" event={"ID":"1c271a6d-5a7a-4ca5-a273-845c84488ef5","Type":"ContainerDied","Data":"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d"} Nov 04 01:28:44.860073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.859916 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7" event={"ID":"1c271a6d-5a7a-4ca5-a273-845c84488ef5","Type":"ContainerDied","Data":"c3c8c0b83e322b29f2fb09a3875af40a19667a9656a73220d55a68c99e768478"} Nov 04 01:28:44.861952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.861926 4486 generic.go:356] "Generic (PLEG): container finished" podID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerID="3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a" exitCode=0 Nov 04 01:28:44.862126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.862017 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" event={"ID":"af563074-bd2d-4b27-a57d-0a1f9d26b186","Type":"ContainerDied","Data":"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a"} Nov 04 01:28:44.862126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.862056 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" event={"ID":"af563074-bd2d-4b27-a57d-0a1f9d26b186","Type":"ContainerDied","Data":"ff270a2213068a1f67c1dc31f30cf94d0d20720bf64dfccc40b099b51c191e30"} Nov 04 01:28:44.862253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.862129 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp" Nov 04 01:28:44.875419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.875396 4486 scope.go:122] "RemoveContainer" containerID="c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b" Nov 04 01:28:44.875877 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:44.875851 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b\": container with ID starting with c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b not found: ID does not exist" containerID="c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b" Nov 04 01:28:44.876002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.875888 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b"} err="failed to get container status \"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b\": rpc error: code = NotFound desc = could not find container \"c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b\": container with ID starting with c651f2de8766d7e49c009939ecb812bfef6bbc95c9876458578fb48ddca67e5b not found: ID does not exist" Nov 04 01:28:44.876002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.875911 4486 scope.go:122] "RemoveContainer" containerID="95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d" Nov 04 01:28:44.894286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.894263 4486 scope.go:122] "RemoveContainer" containerID="95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d" Nov 04 01:28:44.894633 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:44.894609 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d\": container with ID starting with 95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d not found: ID does not exist" containerID="95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d" Nov 04 01:28:44.894717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.894641 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d"} err="failed to get container status \"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d\": rpc error: code = NotFound desc = could not find container \"95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d\": container with ID starting with 95f1a43e7cb6fbc43408f134c39d0fafe6c49f6f054da426866a974516f8635d not found: ID does not exist" Nov 04 01:28:44.894717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.894664 4486 scope.go:122] "RemoveContainer" containerID="3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a" Nov 04 01:28:44.907435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.907408 4486 scope.go:122] "RemoveContainer" containerID="3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a" Nov 04 01:28:44.908419 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:28:44.908391 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a\": container with ID starting with 3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a not found: ID does not exist" containerID="3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a" Nov 04 01:28:44.908537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:44.908427 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a"} err="failed to get container status \"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a\": rpc error: code = NotFound desc = could not find container \"3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a\": container with ID starting with 3857ad2ba8a3e01e90a22d2eef81e4c1eec27795f5dea88c48f1e27de4b0177a not found: ID does not exist" Nov 04 01:28:45.033119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.033085 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:28:45.045892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.045856 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-m6w2s"] Nov 04 01:28:45.152334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.152253 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:28:45.156233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.156206 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-lpnpp"] Nov 04 01:28:45.258797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.258765 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:28:45.268088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.268062 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2ee045-openshift-default-65756bfbc7-jp9v7"] Nov 04 01:28:45.313313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313289 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2"] Nov 04 01:28:45.313898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313884 4486 cpu_manager.go:414] "containerMap: removing container" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313899 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313913 4486 cpu_manager.go:414] "containerMap: removing container" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313918 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313931 4486 cpu_manager.go:414] "containerMap: removing container" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313936 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.313947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313946 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313951 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313958 4486 cpu_manager.go:414] "containerMap: removing container" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313963 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313969 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.313973 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314000 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314007 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314025 4486 cpu_manager.go:414] "containerMap: removing container" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314029 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314036 4486 cpu_manager.go:414] "containerMap: removing container" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314041 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314047 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314052 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314060 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314065 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314080 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314084 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314172 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.314174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314181 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b106dfc4-ef90-4c74-a481-91c119cf4c4c" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314188 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="94101ccd-0f20-4e5c-8226-7adb50643c66" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314196 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c58f026-1d30-4188-9bb0-251c1a968b20" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314202 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314209 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="545a415b-07e6-425b-a489-f269781812f2" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314215 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314221 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d93c59d3-186c-4e77-a8ae-40c1e4102cc4" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314226 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314232 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" containerName="istio-proxy" Nov 04 01:28:45.314739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.314240 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" containerName="istio-proxy" Nov 04 01:28:45.320221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.320205 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.350955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.350928 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07be9abf-2036-4221-805a-b1b856ac1be3" path="/var/lib/kubelet/pods/07be9abf-2036-4221-805a-b1b856ac1be3/volumes" Nov 04 01:28:45.351630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.351606 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c271a6d-5a7a-4ca5-a273-845c84488ef5" path="/var/lib/kubelet/pods/1c271a6d-5a7a-4ca5-a273-845c84488ef5/volumes" Nov 04 01:28:45.351767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.351634 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.351767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.351706 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnxk\" (UniqueName: \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.352147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.352117 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.352289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.352260 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545a415b-07e6-425b-a489-f269781812f2" path="/var/lib/kubelet/pods/545a415b-07e6-425b-a489-f269781812f2/volumes" Nov 04 01:28:45.353419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.353398 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c" path="/var/lib/kubelet/pods/7c05fa5e-ecc6-4d08-8cec-b23ecc3b5e4c/volumes" Nov 04 01:28:45.354134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.354114 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af563074-bd2d-4b27-a57d-0a1f9d26b186" path="/var/lib/kubelet/pods/af563074-bd2d-4b27-a57d-0a1f9d26b186/volumes" Nov 04 01:28:45.354839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.354821 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3891318-4bb2-4c56-bea8-f6a314fe7fcb" path="/var/lib/kubelet/pods/f3891318-4bb2-4c56-bea8-f6a314fe7fcb/volumes" Nov 04 01:28:45.374670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.374635 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:28:45.374819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.374679 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:28:45.374819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.374709 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2"] Nov 04 01:28:45.380510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.380480 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4"] Nov 04 01:28:45.381339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.381320 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="92966232-2a3d-4a7e-896c-c77096e8fdd1" containerName="istio-proxy" Nov 04 01:28:45.386166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.386146 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.436188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.436107 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4"] Nov 04 01:28:45.453305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453268 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.453490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453363 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnxk\" (UniqueName: \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.453641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453490 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jrj\" (UniqueName: \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.453641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453529 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.453641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453567 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.453921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.453896 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.454355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.454327 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.456268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.456243 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.511403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.511371 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnxk\" (UniqueName: \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\") pod \"maas-api-revoke-sub-f1b61114-mjfs2\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.554921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.554893 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.555128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.555110 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jrj\" (UniqueName: \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.555190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.555141 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.555781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.555758 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.558081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.558054 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.610913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.610882 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jrj\" (UniqueName: \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\") pod \"maas-api-revoke-sub-b5af534c-nzbp4\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:45.634612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.634593 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:45.716404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:45.716162 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:46.009672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.009642 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2"] Nov 04 01:28:46.011128 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:46.011101 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a076ce_769e_47fa_a775_a5bff8622f77.slice/crio-06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869 WatchSource:0}: Error finding container 06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869: Status 404 returned error can't find the container with id 06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869 Nov 04 01:28:46.112574 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:46.112537 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c616ad_ef19_4989_9d7f_55e2dff35fd0.slice/crio-e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212 WatchSource:0}: Error finding container e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212: Status 404 returned error can't find the container with id e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212 Nov 04 01:28:46.113334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.113302 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4"] Nov 04 01:28:46.887826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.887792 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" event={"ID":"93a076ce-769e-47fa-a775-a5bff8622f77","Type":"ContainerStarted","Data":"3e5cc98b46ce1f8f4d6b9590f36cf06ba30950a0aa362ebd2897532ff959cfe7"} Nov 04 01:28:46.888051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.887855 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" event={"ID":"93a076ce-769e-47fa-a775-a5bff8622f77","Type":"ContainerStarted","Data":"06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869"} Nov 04 01:28:46.889731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.889701 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" event={"ID":"a3c616ad-ef19-4989-9d7f-55e2dff35fd0","Type":"ContainerStarted","Data":"ce916aca4142a3077a60c5431f9c6c192e23245e4cb83a12d2d3aae1a94f8f04"} Nov 04 01:28:46.889835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:46.889751 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" event={"ID":"a3c616ad-ef19-4989-9d7f-55e2dff35fd0","Type":"ContainerStarted","Data":"e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212"} Nov 04 01:28:47.002482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.002431 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" podStartSLOduration=2.00241704 podStartE2EDuration="2.00241704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.998158995 +0000 UTC m=+1895.756821158" watchObservedRunningTime="2026-11-04 01:28:47.00241704 +0000 UTC m=+1895.761079242" Nov 04 01:28:47.114582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.114515 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" podStartSLOduration=2.1145005550000002 podStartE2EDuration="2.114500555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:47.108595821 +0000 UTC m=+1895.867258001" watchObservedRunningTime="2026-11-04 01:28:47.114500555 +0000 UTC m=+1895.873162708" Nov 04 01:28:47.901522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.901483 4486 generic.go:356] "Generic (PLEG): container finished" podID="a3c616ad-ef19-4989-9d7f-55e2dff35fd0" containerID="ce916aca4142a3077a60c5431f9c6c192e23245e4cb83a12d2d3aae1a94f8f04" exitCode=0 Nov 04 01:28:47.901686 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.901581 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" event={"ID":"a3c616ad-ef19-4989-9d7f-55e2dff35fd0","Type":"ContainerDied","Data":"ce916aca4142a3077a60c5431f9c6c192e23245e4cb83a12d2d3aae1a94f8f04"} Nov 04 01:28:47.903691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.903662 4486 generic.go:356] "Generic (PLEG): container finished" podID="93a076ce-769e-47fa-a775-a5bff8622f77" containerID="3e5cc98b46ce1f8f4d6b9590f36cf06ba30950a0aa362ebd2897532ff959cfe7" exitCode=0 Nov 04 01:28:47.903850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:47.903748 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" event={"ID":"93a076ce-769e-47fa-a775-a5bff8622f77","Type":"ContainerDied","Data":"3e5cc98b46ce1f8f4d6b9590f36cf06ba30950a0aa362ebd2897532ff959cfe7"} Nov 04 01:28:49.085473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.085445 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:49.140420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.140393 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:49.211320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211246 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\") pod \"93a076ce-769e-47fa-a775-a5bff8622f77\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " Nov 04 01:28:49.211320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211310 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\") pod \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " Nov 04 01:28:49.211548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211493 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\") pod \"93a076ce-769e-47fa-a775-a5bff8622f77\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " Nov 04 01:28:49.211632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211616 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\" (UniqueName: \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\") pod \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " Nov 04 01:28:49.211722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211696 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\" (UniqueName: \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\") pod \"93a076ce-769e-47fa-a775-a5bff8622f77\" (UID: \"93a076ce-769e-47fa-a775-a5bff8622f77\") " Nov 04 01:28:49.211846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.211755 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\") pod \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\" (UID: \"a3c616ad-ef19-4989-9d7f-55e2dff35fd0\") " Nov 04 01:28:49.213143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.213117 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca" pod "93a076ce-769e-47fa-a775-a5bff8622f77" (UID: "93a076ce-769e-47fa-a775-a5bff8622f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:49.213934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.213897 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca" pod "93a076ce-769e-47fa-a775-a5bff8622f77" (UID: "93a076ce-769e-47fa-a775-a5bff8622f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:49.214088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.213971 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca" pod "a3c616ad-ef19-4989-9d7f-55e2dff35fd0" (UID: "a3c616ad-ef19-4989-9d7f-55e2dff35fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:28:49.214330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.214309 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca" pod "a3c616ad-ef19-4989-9d7f-55e2dff35fd0" (UID: "a3c616ad-ef19-4989-9d7f-55e2dff35fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:28:49.215072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.215045 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj" pod "a3c616ad-ef19-4989-9d7f-55e2dff35fd0" (UID: "a3c616ad-ef19-4989-9d7f-55e2dff35fd0"). InnerVolumeSpecName "kube-api-access-44jrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:49.216804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.216778 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk" pod "93a076ce-769e-47fa-a775-a5bff8622f77" (UID: "93a076ce-769e-47fa-a775-a5bff8622f77"). InnerVolumeSpecName "kube-api-access-bgnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:28:49.313742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313706 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.313742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313735 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jrj\" (UniqueName: \"kubernetes.io/projected/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-kube-api-access-44jrj\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.313742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313747 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnxk\" (UniqueName: \"kubernetes.io/projected/93a076ce-769e-47fa-a775-a5bff8622f77-kube-api-access-bgnxk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.314051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313756 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.314051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313767 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a076ce-769e-47fa-a775-a5bff8622f77-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.314051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.313776 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c616ad-ef19-4989-9d7f-55e2dff35fd0-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:28:49.924084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.924050 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" Nov 04 01:28:49.924270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.924046 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2" event={"ID":"93a076ce-769e-47fa-a775-a5bff8622f77","Type":"ContainerDied","Data":"06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869"} Nov 04 01:28:49.924270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.924198 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06681dc46644e8bc94edbcbd31497e885e82ef1e37bf897eaeab66456304a869" Nov 04 01:28:49.925577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.925558 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" Nov 04 01:28:49.925696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.925609 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4" event={"ID":"a3c616ad-ef19-4989-9d7f-55e2dff35fd0","Type":"ContainerDied","Data":"e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212"} Nov 04 01:28:49.925696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:49.925653 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e223a0eb2d8ed650f1c303e3049ef581b5867431da51c1654a3cad3b93eb6212" Nov 04 01:28:52.787601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:52.787565 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl"] Nov 04 01:28:52.836661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:52.836628 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8aa9f3a2-vvpgl"] Nov 04 01:28:53.352462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:53.352432 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49648e5-fc2e-4c81-96d8-3c542b04ad81" path="/var/lib/kubelet/pods/d49648e5-fc2e-4c81-96d8-3c542b04ad81/volumes" Nov 04 01:28:59.178016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.177965 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:28:59.178665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178648 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a3c616ad-ef19-4989-9d7f-55e2dff35fd0" containerName="revoke-keys" Nov 04 01:28:59.178712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178666 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c616ad-ef19-4989-9d7f-55e2dff35fd0" containerName="revoke-keys" Nov 04 01:28:59.178712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178682 4486 cpu_manager.go:414] "containerMap: removing container" podUID="93a076ce-769e-47fa-a775-a5bff8622f77" containerName="revoke-keys" Nov 04 01:28:59.178712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178687 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a076ce-769e-47fa-a775-a5bff8622f77" containerName="revoke-keys" Nov 04 01:28:59.178812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178803 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c616ad-ef19-4989-9d7f-55e2dff35fd0" containerName="revoke-keys" Nov 04 01:28:59.178852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.178815 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a076ce-769e-47fa-a775-a5bff8622f77" containerName="revoke-keys" Nov 04 01:28:59.183726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.183701 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.244858 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.244810 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:28:59.280727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.280685 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-98f846-openshift-default-dockercfg-drblk\"" Nov 04 01:28:59.290481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290449 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290648 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290620 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290667 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290753 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290791 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290851 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24l9w\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290917 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.290972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.290954 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.291028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.291019 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.393866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.393814 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394046 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394078 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394139 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394180 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394249 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24l9w\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394304 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394375 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394426 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.394880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.394858 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.396874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.396847 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.397082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.397065 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.397569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.397547 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.397712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.397606 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.398761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.398738 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.400231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.400210 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.449751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.449678 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.450570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.450541 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24l9w\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\") pod \"e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.498335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.498303 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:28:59.943962 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:28:59.943925 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3af9229_7b9f_4553_8896_b66f6a75b146.slice/crio-0ed189b748605a69e15385f9a79ed443064adbc69ad26b9a319c1fae80f7e8c1 WatchSource:0}: Error finding container 0ed189b748605a69e15385f9a79ed443064adbc69ad26b9a319c1fae80f7e8c1: Status 404 returned error can't find the container with id 0ed189b748605a69e15385f9a79ed443064adbc69ad26b9a319c1fae80f7e8c1 Nov 04 01:28:59.945497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.945471 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:28:59.947264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.947230 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:59.947378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.947295 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:28:59.947378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:28:59.947325 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:00.055218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.054472 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerStarted","Data":"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb"} Nov 04 01:29:00.055218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.054537 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerStarted","Data":"0ed189b748605a69e15385f9a79ed443064adbc69ad26b9a319c1fae80f7e8c1"} Nov 04 01:29:00.174304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.174238 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podStartSLOduration=1.174219249 podStartE2EDuration="1.174219249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:00.171105148 +0000 UTC m=+1908.929767309" watchObservedRunningTime="2026-11-04 01:29:00.174219249 +0000 UTC m=+1908.932881411" Nov 04 01:29:00.499129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.499093 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:29:00.799151 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.799069 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:29:00.799629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:00.799330 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" containerID="cri-o://ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07" gracePeriod=30 Nov 04 01:29:01.504970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:01.504940 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:29:02.095070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.095034 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:29:02.096065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.096039 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:29:02.260593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.260556 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf"] Nov 04 01:29:02.265544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.265519 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.317671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.315371 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:29:02.317671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.315471 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:29:02.320240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.320211 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf"] Nov 04 01:29:02.475527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.475420 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfjh\" (UniqueName: \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.475527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.475484 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.475762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.475613 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.576860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.576828 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfjh\" (UniqueName: \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.577371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.576872 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.577371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.576960 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.577600 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.577577 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.579515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.579489 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.631995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.631954 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfjh\" (UniqueName: \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:02.883063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:02.883030 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:03.268475 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:29:03.268442 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ae0587_f0b1_47dc_a8de_7cd89c40729d.slice/crio-f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73 WatchSource:0}: Error finding container f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73: Status 404 returned error can't find the container with id f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73 Nov 04 01:29:03.269244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:03.269218 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf"] Nov 04 01:29:04.112782 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:04.112746 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" event={"ID":"33ae0587-f0b1-47dc-a8de-7cd89c40729d","Type":"ContainerStarted","Data":"e8fb46f0fdab6453669174a97bb19f5533cc451b34aa4ca2e3b2245aad224dd4"} Nov 04 01:29:04.113212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:04.112806 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" event={"ID":"33ae0587-f0b1-47dc-a8de-7cd89c40729d","Type":"ContainerStarted","Data":"f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73"} Nov 04 01:29:04.225825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:04.225770 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" podStartSLOduration=2.225755134 podStartE2EDuration="2.225755134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:04.223328069 +0000 UTC m=+1912.981990230" watchObservedRunningTime="2026-11-04 01:29:04.225755134 +0000 UTC m=+1912.984417298" Nov 04 01:29:05.124725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:05.124689 4486 generic.go:356] "Generic (PLEG): container finished" podID="33ae0587-f0b1-47dc-a8de-7cd89c40729d" containerID="e8fb46f0fdab6453669174a97bb19f5533cc451b34aa4ca2e3b2245aad224dd4" exitCode=0 Nov 04 01:29:05.125203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:05.124778 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" event={"ID":"33ae0587-f0b1-47dc-a8de-7cd89c40729d","Type":"ContainerDied","Data":"e8fb46f0fdab6453669174a97bb19f5533cc451b34aa4ca2e3b2245aad224dd4"} Nov 04 01:29:06.129612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.129584 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:29:06.134417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.134392 4486 generic.go:356] "Generic (PLEG): container finished" podID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerID="ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07" exitCode=0 Nov 04 01:29:06.134522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.134442 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" Nov 04 01:29:06.134522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.134438 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerDied","Data":"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07"} Nov 04 01:29:06.134626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.134564 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq" event={"ID":"b92bf487-dd47-4d52-8aa9-9258f2ba92d0","Type":"ContainerDied","Data":"b846d22a63d2a6faded116b123ad557502e8a19472217745d4affb129e231a13"} Nov 04 01:29:06.134626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.134591 4486 scope.go:122] "RemoveContainer" containerID="ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07" Nov 04 01:29:06.145892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.145866 4486 scope.go:122] "RemoveContainer" containerID="73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95" Nov 04 01:29:06.156657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.156636 4486 scope.go:122] "RemoveContainer" containerID="ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07" Nov 04 01:29:06.156960 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:29:06.156930 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07\": container with ID starting with ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07 not found: ID does not exist" containerID="ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07" Nov 04 01:29:06.157108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.156968 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07"} err="failed to get container status \"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07\": rpc error: code = NotFound desc = could not find container \"ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07\": container with ID starting with ef774a2938068bcd29db8ead71bde6f03fd3cd7e61b8433f823c4fe2c6663d07 not found: ID does not exist" Nov 04 01:29:06.157108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.157000 4486 scope.go:122] "RemoveContainer" containerID="73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95" Nov 04 01:29:06.157228 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:29:06.157209 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95\": container with ID starting with 73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95 not found: ID does not exist" containerID="73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95" Nov 04 01:29:06.157275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.157227 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95"} err="failed to get container status \"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95\": rpc error: code = NotFound desc = could not find container \"73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95\": container with ID starting with 73cd8ed85be2f68b1ac6b102c1abbbf761f55dd4847dc7c4a6798b841888ec95 not found: ID does not exist" Nov 04 01:29:06.241348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241059 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.241348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241140 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.241348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241175 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.241348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241271 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.241348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241325 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.241731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241383 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.242065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.241812 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:06.242065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242012 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:06.242065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242023 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:06.242656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242054 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.242656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242080 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:06.242656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242136 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.242656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242258 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\") pod \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\" (UID: \"b92bf487-dd47-4d52-8aa9-9258f2ba92d0\") " Nov 04 01:29:06.242656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.242331 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:06.243196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.243160 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.243372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.243355 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.245861 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.245823 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.246024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.246007 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.246501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.246483 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.253736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.249741 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:06.254654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.254629 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:06.254852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.254831 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:06.256297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.256270 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw" pod "b92bf487-dd47-4d52-8aa9-9258f2ba92d0" (UID: "b92bf487-dd47-4d52-8aa9-9258f2ba92d0"). InnerVolumeSpecName "kube-api-access-4gsgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:06.279789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.279768 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:06.347355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.347261 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\") pod \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " Nov 04 01:29:06.347532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.347410 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\" (UniqueName: \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\") pod \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " Nov 04 01:29:06.347532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.347430 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\") pod \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\" (UID: \"33ae0587-f0b1-47dc-a8de-7cd89c40729d\") " Nov 04 01:29:06.347695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.347653 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca" pod "33ae0587-f0b1-47dc-a8de-7cd89c40729d" (UID: "33ae0587-f0b1-47dc-a8de-7cd89c40729d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:06.348109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.348081 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.348208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.348113 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.348208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.348129 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.348208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.348145 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gsgw\" (UniqueName: \"kubernetes.io/projected/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-kube-api-access-4gsgw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.348208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.348160 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92bf487-dd47-4d52-8aa9-9258f2ba92d0-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.349700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.349678 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh" pod "33ae0587-f0b1-47dc-a8de-7cd89c40729d" (UID: "33ae0587-f0b1-47dc-a8de-7cd89c40729d"). InnerVolumeSpecName "kube-api-access-gpfjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:06.350191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.350172 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca" pod "33ae0587-f0b1-47dc-a8de-7cd89c40729d" (UID: "33ae0587-f0b1-47dc-a8de-7cd89c40729d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:06.449750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.449708 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpfjh\" (UniqueName: \"kubernetes.io/projected/33ae0587-f0b1-47dc-a8de-7cd89c40729d-kube-api-access-gpfjh\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.449750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.449737 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ae0587-f0b1-47dc-a8de-7cd89c40729d-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:06.637710 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.637626 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:29:06.644096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:06.644062 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8aa9f3a2-openshift-default-587f8d6969-k85cq"] Nov 04 01:29:07.145586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:07.145554 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" Nov 04 01:29:07.145977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:07.145590 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf" event={"ID":"33ae0587-f0b1-47dc-a8de-7cd89c40729d","Type":"ContainerDied","Data":"f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73"} Nov 04 01:29:07.145977 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:07.145644 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a7b9294735c0a4d10ddbfda710efe7c32bd127187c790dd744434147037d73" Nov 04 01:29:07.350624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:07.350587 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" path="/var/lib/kubelet/pods/b92bf487-dd47-4d52-8aa9-9258f2ba92d0/volumes" Nov 04 01:29:13.832466 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:29:13.832423 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:29:14.286631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:14.286506 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:29:15.047183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:15.047142 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76"] Nov 04 01:29:15.055591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:15.055501 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b406df7-rtg76"] Nov 04 01:29:15.355885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:15.355846 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e723bb5-36fd-41d1-a160-4169b8fbbb7f" path="/var/lib/kubelet/pods/1e723bb5-36fd-41d1-a160-4169b8fbbb7f/volumes" Nov 04 01:29:19.287271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.287233 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" podUID="da6c003d-ee8f-4685-959b-444edc0d973b" containerName="payload-pre-processing" containerID="cri-o://e2d59cceb5db1e02cd83465b19fc11017b8feed40705d8aac76d5916d72340ff" gracePeriod=25 Nov 04 01:29:19.572338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.572310 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:29:19.691794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.690647 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\") pod \"da6c003d-ee8f-4685-959b-444edc0d973b\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " Nov 04 01:29:19.691794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.690878 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\" (UniqueName: \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\") pod \"da6c003d-ee8f-4685-959b-444edc0d973b\" (UID: \"da6c003d-ee8f-4685-959b-444edc0d973b\") " Nov 04 01:29:19.691794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.691153 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume" pod "da6c003d-ee8f-4685-959b-444edc0d973b" (UID: "da6c003d-ee8f-4685-959b-444edc0d973b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:19.691794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.691531 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c003d-ee8f-4685-959b-444edc0d973b-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:19.694109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.694084 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr" pod "da6c003d-ee8f-4685-959b-444edc0d973b" (UID: "da6c003d-ee8f-4685-959b-444edc0d973b"). InnerVolumeSpecName "kube-api-access-xmnlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:19.792867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.792780 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmnlr\" (UniqueName: \"kubernetes.io/projected/da6c003d-ee8f-4685-959b-444edc0d973b-kube-api-access-xmnlr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:19.794994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.794965 4486 scope.go:122] "RemoveContainer" containerID="e2d59cceb5db1e02cd83465b19fc11017b8feed40705d8aac76d5916d72340ff" Nov 04 01:29:19.806364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:19.806344 4486 scope.go:122] "RemoveContainer" containerID="820e3b81ac63b0fb586436936ce130eb4190a9819bb940f213dbccfdf06d40f7" Nov 04 01:29:20.264348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:20.264313 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" event={"ID":"da6c003d-ee8f-4685-959b-444edc0d973b","Type":"ContainerDied","Data":"e2d59cceb5db1e02cd83465b19fc11017b8feed40705d8aac76d5916d72340ff"} Nov 04 01:29:20.264538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:20.264376 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" event={"ID":"da6c003d-ee8f-4685-959b-444edc0d973b","Type":"ContainerDied","Data":"c7593fd6534630293d0fa83731e12d7cc4db9740dcf694836256976c1af69166"} Nov 04 01:29:20.264538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:20.264385 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn" Nov 04 01:29:20.497291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:20.497253 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:29:20.501640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:20.501611 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e65b23-7c5f5d6bf9-msmdn"] Nov 04 01:29:21.349521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:21.349481 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6c003d-ee8f-4685-959b-444edc0d973b" path="/var/lib/kubelet/pods/da6c003d-ee8f-4685-959b-444edc0d973b/volumes" Nov 04 01:29:43.885144 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:29:43.885077 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:29:48.751193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:48.751148 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:48.751697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:48.751226 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:52.086012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:52.085959 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j"] Nov 04 01:29:52.131306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:52.131225 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c563ee-rg96j"] Nov 04 01:29:53.350143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.350107 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b904f7-dbf3-4fc5-bc58-7fe693102882" path="/var/lib/kubelet/pods/b2b904f7-dbf3-4fc5-bc58-7fe693102882/volumes" Nov 04 01:29:53.671695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.671612 4486 generic.go:356] "Generic (PLEG): container finished" podID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerID="6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564" exitCode=0 Nov 04 01:29:53.671848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.671690 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerDied","Data":"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564"} Nov 04 01:29:53.672244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.672206 4486 scope.go:122] "RemoveContainer" containerID="6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564" Nov 04 01:29:53.673527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.673508 4486 generic.go:356] "Generic (PLEG): container finished" podID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerID="78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925" exitCode=0 Nov 04 01:29:53.673605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.673584 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerDied","Data":"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925"} Nov 04 01:29:53.674070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.674052 4486 scope.go:122] "RemoveContainer" containerID="78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925" Nov 04 01:29:53.674121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.674074 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:53.674208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.674191 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:53.674264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.674250 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:53.675564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.675537 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:53.675658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.675599 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:53.675658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:53.675631 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:29:54.689332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:54.689294 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerStarted","Data":"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990"} Nov 04 01:29:54.692015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:54.691966 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerStarted","Data":"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64"} Nov 04 01:29:54.910007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:54.909472 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:29:54.927582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:54.913143 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.927582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:54.913287 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:29:55.014063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.002649 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:29:55.014063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.002785 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.014063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.002826 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:29:55.902902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.902847 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.903341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.902927 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:29:55.993451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.993407 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.993640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:55.993462 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:29:56.612816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.612777 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c"] Nov 04 01:29:56.613466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613448 4486 cpu_manager.go:414] "containerMap: removing container" podUID="da6c003d-ee8f-4685-959b-444edc0d973b" containerName="payload-pre-processing" Nov 04 01:29:56.613532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613467 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6c003d-ee8f-4685-959b-444edc0d973b" containerName="payload-pre-processing" Nov 04 01:29:56.613532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613480 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.613532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613486 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.613532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613512 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.613532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613519 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.613697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613561 4486 cpu_manager.go:414] "containerMap: removing container" podUID="33ae0587-f0b1-47dc-a8de-7cd89c40729d" containerName="revoke-keys" Nov 04 01:29:56.613697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613566 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ae0587-f0b1-47dc-a8de-7cd89c40729d" containerName="revoke-keys" Nov 04 01:29:56.613697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613677 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.613697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613693 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ae0587-f0b1-47dc-a8de-7cd89c40729d" containerName="revoke-keys" Nov 04 01:29:56.613828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.613700 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6c003d-ee8f-4685-959b-444edc0d973b" containerName="payload-pre-processing" Nov 04 01:29:56.616483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.616464 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.691591 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf"] Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.692752 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92bf487-dd47-4d52-8aa9-9258f2ba92d0" containerName="istio-proxy" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.706180 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.706281 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.706345 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s776h\" (UniqueName: \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.712033 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:29:56.722066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.712649 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:29:56.727441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.727415 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:56.780638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.780593 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c"] Nov 04 01:29:56.808923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.808884 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.809277 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.809254 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.809454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.809436 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s776h\" (UniqueName: \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.811677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.811649 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.814516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.814295 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.817721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.817702 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf"] Nov 04 01:29:56.917687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.917599 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zt5\" (UniqueName: \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:56.918198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.917720 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.918198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.917783 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:56.918198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.917785 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:29:56.918198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.917963 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:56.919260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.919236 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s776h\" (UniqueName: \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\") pod \"maas-api-revoke-sub-87d9f19a-xzl2c\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:56.988020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:56.980379 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:29:57.009591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.009547 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.009763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.009658 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:29:57.019498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.019463 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zt5\" (UniqueName: \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.019651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.019597 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.019716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.019703 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.020452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.020422 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.022331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.022304 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.094164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.094096 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zt5\" (UniqueName: \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\") pod \"maas-api-revoke-sub-e1b8051b-x9jvf\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.220425 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.220390 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:29:57.502774 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:29:57.502710 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4731bd77_09b8_4ab5_b439_8670affb5d57.slice/crio-6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543 WatchSource:0}: Error finding container 6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543: Status 404 returned error can't find the container with id 6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543 Nov 04 01:29:57.504356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.504325 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c"] Nov 04 01:29:57.710489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.710440 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf"] Nov 04 01:29:57.806540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.806507 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" event={"ID":"4731bd77-09b8-4ab5-b439-8670affb5d57","Type":"ContainerStarted","Data":"6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543"} Nov 04 01:29:57.904110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.903609 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.904110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.903667 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:29:57.993697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.993664 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.994087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:57.993713 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:29:58.819480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.819448 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" event={"ID":"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7","Type":"ContainerStarted","Data":"def09d85df9c38542696d36618cf2aa4317ea37ab86486440f4ff74d9a17747d"} Nov 04 01:29:58.819631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.819511 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" event={"ID":"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7","Type":"ContainerStarted","Data":"3a4f1ddcd7f048b090c6f0c6aa6307ab60c64ecf1c9604da9056d7dcc279ec70"} Nov 04 01:29:58.821286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.821260 4486 generic.go:356] "Generic (PLEG): container finished" podID="4731bd77-09b8-4ab5-b439-8670affb5d57" containerID="d8d1c27284326041055dea58f9a93b009ae0af6451704860a9df96c77bfaeece" exitCode=0 Nov 04 01:29:58.821414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.821342 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" event={"ID":"4731bd77-09b8-4ab5-b439-8670affb5d57","Type":"ContainerDied","Data":"d8d1c27284326041055dea58f9a93b009ae0af6451704860a9df96c77bfaeece"} Nov 04 01:29:58.905038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.903301 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.905038 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.903362 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:29:58.963973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.963908 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" podStartSLOduration=2.963887616 podStartE2EDuration="2.963887616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:58.956306384 +0000 UTC m=+1967.714968545" watchObservedRunningTime="2026-11-04 01:29:58.963887616 +0000 UTC m=+1967.722549780" Nov 04 01:29:59.000209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.999622 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.000209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:58.999689 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:29:59.215829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.215784 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:29:59.216121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.216086 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" containerID="cri-o://d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c" gracePeriod=30 Nov 04 01:29:59.219228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.219199 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:29:59.219522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.219497 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" containerID="cri-o://69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1" gracePeriod=30 Nov 04 01:29:59.224402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.224380 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:29:59.224657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.224634 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" containerID="cri-o://b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995" gracePeriod=30 Nov 04 01:29:59.618808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.618775 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:29:59.900817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.900729 4486 generic.go:356] "Generic (PLEG): container finished" podID="2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" containerID="def09d85df9c38542696d36618cf2aa4317ea37ab86486440f4ff74d9a17747d" exitCode=0 Nov 04 01:29:59.901033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.901007 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" event={"ID":"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7","Type":"ContainerDied","Data":"def09d85df9c38542696d36618cf2aa4317ea37ab86486440f4ff74d9a17747d"} Nov 04 01:29:59.902973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.902941 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.903120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:29:59.903022 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 04 01:30:00.009006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.008114 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 04 01:30:00.009006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.008176 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 04 01:30:00.179537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.179508 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:30:00.223006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.222183 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:00.230808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.223359 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4731bd77-09b8-4ab5-b439-8670affb5d57" containerName="revoke-keys" Nov 04 01:30:00.230808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.223387 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4731bd77-09b8-4ab5-b439-8670affb5d57" containerName="revoke-keys" Nov 04 01:30:00.230808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.223553 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4731bd77-09b8-4ab5-b439-8670affb5d57" containerName="revoke-keys" Nov 04 01:30:00.230808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.226768 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.231176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.231067 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:00.240482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.238685 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.245029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.245000 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:00.258511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.258354 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:00.259023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.258997 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.262909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.262890 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:00.263133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.263034 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.267162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.267142 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.274190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.274166 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.306633 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\") pod \"4731bd77-09b8-4ab5-b439-8670affb5d57\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.306742 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\") pod \"4731bd77-09b8-4ab5-b439-8670affb5d57\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.306894 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\" (UniqueName: \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\") pod \"4731bd77-09b8-4ab5-b439-8670affb5d57\" (UID: \"4731bd77-09b8-4ab5-b439-8670affb5d57\") " Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.307554 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca" pod "4731bd77-09b8-4ab5-b439-8670affb5d57" (UID: "4731bd77-09b8-4ab5-b439-8670affb5d57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.307564 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.307755 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpr6\" (UniqueName: \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.307868 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbt2\" (UniqueName: \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.307927 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.314043 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.314101 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.314222 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.320043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.317286 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h" pod "4731bd77-09b8-4ab5-b439-8670affb5d57" (UID: "4731bd77-09b8-4ab5-b439-8670affb5d57"). InnerVolumeSpecName "kube-api-access-s776h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:00.376463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.339365 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:00.376463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.370016 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:00.376463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.370118 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:00.376463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.374948 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca" pod "4731bd77-09b8-4ab5-b439-8670affb5d57" (UID: "4731bd77-09b8-4ab5-b439-8670affb5d57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:00.380330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.378605 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:00.386768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.386721 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:00.387444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.387244 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.402499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.402463 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:00.402698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.402568 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.464298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.463792 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpr6\" (UniqueName: \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.468125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.468095 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.468407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.468386 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.468582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.468566 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g82n\" (UniqueName: \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.468753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.468735 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbt2\" (UniqueName: \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.469171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.469150 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.469343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.469325 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdj7\" (UniqueName: \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.469512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.469495 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.469727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.469707 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.469901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.469881 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmvb\" (UniqueName: \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.470077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470059 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98bc\" (UniqueName: \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.470206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470189 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.470429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470409 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s776h\" (UniqueName: \"kubernetes.io/projected/4731bd77-09b8-4ab5-b439-8670affb5d57-kube-api-access-s776h\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:00.470546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470531 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4731bd77-09b8-4ab5-b439-8670affb5d57-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:00.470682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470073 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.470788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.470747 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.471726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.471669 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:00.479385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.479356 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:00.488888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.488846 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:00.532240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.531263 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpr6\" (UniqueName: \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\") pod \"maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.532440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.532268 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbt2\" (UniqueName: \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.563413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.563378 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.576976 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g82n\" (UniqueName: \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577110 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577142 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdj7\" (UniqueName: \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577193 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrj9\" (UniqueName: \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577259 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577414 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmvb\" (UniqueName: \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577456 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98bc\" (UniqueName: \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577491 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtw5p\" (UniqueName: \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577531 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577598 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.577705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577633 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.578443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.577717 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.579054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.578892 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.580953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.579867 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.580953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.580485 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.581226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.581070 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.613244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.613209 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:00.639020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.637690 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdj7\" (UniqueName: \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\") pod \"maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.639020 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.638955 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g82n\" (UniqueName: \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.643666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.643641 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmvb\" (UniqueName: \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.647443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.647412 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98bc\" (UniqueName: \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\") pod \"maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.673440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.673163 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:00.687006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.686208 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtw5p\" (UniqueName: \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.687006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.686328 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.687006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.686419 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.687006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.686471 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrj9\" (UniqueName: \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.687850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.687768 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.687850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.687806 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.688917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.688312 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:00.702348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.702105 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:00.718286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.718111 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:00.781729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.780737 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtw5p\" (UniqueName: \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.781729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.781128 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrj9\" (UniqueName: \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\") pod \"maas-api-key-cleanup-29895930-v6pp8\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:00.818108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.814761 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:00.910744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.909780 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:30:00.910744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.910459 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:30:00.914556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.913068 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:30:00.924427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.924366 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" Nov 04 01:30:00.925490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.924520 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c" event={"ID":"4731bd77-09b8-4ab5-b439-8670affb5d57","Type":"ContainerDied","Data":"6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543"} Nov 04 01:30:00.925556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.925526 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d9e821381a4322db80d40091f4d268730200c4fe1bef9ff6d76c292e585b543" Nov 04 01:30:00.966305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.966268 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:00.966452 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:00.966417 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac273c8_2b61_4570_9a34_eb572bfcedcc.slice/crio-ad4661eb9614f6ff69c0088e1200079813ff5974d51b9af6499c019bb607ea5e WatchSource:0}: Error finding container ad4661eb9614f6ff69c0088e1200079813ff5974d51b9af6499c019bb607ea5e: Status 404 returned error can't find the container with id ad4661eb9614f6ff69c0088e1200079813ff5974d51b9af6499c019bb607ea5e Nov 04 01:30:00.998646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:00.998617 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:30:01.000265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.000243 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:30:01.025799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.025568 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:01.027744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.027718 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:01.027886 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:01.027857 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445d79ac_a030_483b_a248_926b0107bfb9.slice/crio-8db8f500f542be1e50b86b883ce598e5618c34f3f3de862041a1f65ca15195c3 WatchSource:0}: Error finding container 8db8f500f542be1e50b86b883ce598e5618c34f3f3de862041a1f65ca15195c3: Status 404 returned error can't find the container with id 8db8f500f542be1e50b86b883ce598e5618c34f3f3de862041a1f65ca15195c3 Nov 04 01:30:01.146866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.146833 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:01.155547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.154339 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:01.173047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.172754 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:01.181009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.180960 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:01.224703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.224671 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:01.231764 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:01.231723 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefffc5a4_9ba0_4e53_a2c2_b6a6d1ee57ff.slice/crio-65a6594214abe0337a7a022d56f168b5029d6806b06bdaaeca7c44489c1b3423 WatchSource:0}: Error finding container 65a6594214abe0337a7a022d56f168b5029d6806b06bdaaeca7c44489c1b3423: Status 404 returned error can't find the container with id 65a6594214abe0337a7a022d56f168b5029d6806b06bdaaeca7c44489c1b3423 Nov 04 01:30:01.232490 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:01.232459 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9695423d_854a_4fac_a1f8_7185c8eb4a4f.slice/crio-cf495ed5f1b5ec9e174eb80d87a566fb3ef1805b145051704614644feb046329 WatchSource:0}: Error finding container cf495ed5f1b5ec9e174eb80d87a566fb3ef1805b145051704614644feb046329: Status 404 returned error can't find the container with id cf495ed5f1b5ec9e174eb80d87a566fb3ef1805b145051704614644feb046329 Nov 04 01:30:01.242471 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:01.242189 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ccff16_040a_400e_97b2_be728905afa5.slice/crio-890e7a61986b46e8ecf65211a9d432294c35b2cdf3a081c15608e1016e37a6e4 WatchSource:0}: Error finding container 890e7a61986b46e8ecf65211a9d432294c35b2cdf3a081c15608e1016e37a6e4: Status 404 returned error can't find the container with id 890e7a61986b46e8ecf65211a9d432294c35b2cdf3a081c15608e1016e37a6e4 Nov 04 01:30:01.400221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.400190 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:30:01.482838 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:01.482803 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343e4987_1f43_45b6_b654_497177789217.slice/crio-20091b96accbe53b69eb80efd968ba8039c8b1776df90d1349b0f949f9170edd WatchSource:0}: Error finding container 20091b96accbe53b69eb80efd968ba8039c8b1776df90d1349b0f949f9170edd: Status 404 returned error can't find the container with id 20091b96accbe53b69eb80efd968ba8039c8b1776df90d1349b0f949f9170edd Nov 04 01:30:01.484888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.484854 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:01.503770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.503743 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\" (UniqueName: \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\") pod \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " Nov 04 01:30:01.503859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.503794 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\") pod \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " Nov 04 01:30:01.503943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.503925 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\") pod \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\" (UID: \"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7\") " Nov 04 01:30:01.504221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.504192 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca" pod "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" (UID: "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:01.504597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.504574 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.506407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.506385 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5" pod "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" (UID: "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7"). InnerVolumeSpecName "kube-api-access-f2zt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:01.506603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.506578 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca" pod "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" (UID: "2e5221c9-adf8-49d3-90eb-ca45d2bc17b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:01.606731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.606694 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2zt5\" (UniqueName: \"kubernetes.io/projected/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-kube-api-access-f2zt5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.606731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.606729 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:01.939288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.939247 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerStarted","Data":"7bd1ae4b9b7c756e51c2bef1a52a62f86c282bda8227a45292db620104e53f5e"} Nov 04 01:30:01.939398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.939325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerStarted","Data":"8db8f500f542be1e50b86b883ce598e5618c34f3f3de862041a1f65ca15195c3"} Nov 04 01:30:01.949049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.943148 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerStarted","Data":"65a6594214abe0337a7a022d56f168b5029d6806b06bdaaeca7c44489c1b3423"} Nov 04 01:30:01.949049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.945925 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" event={"ID":"2e5221c9-adf8-49d3-90eb-ca45d2bc17b7","Type":"ContainerDied","Data":"3a4f1ddcd7f048b090c6f0c6aa6307ab60c64ecf1c9604da9056d7dcc279ec70"} Nov 04 01:30:01.949049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.945973 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a4f1ddcd7f048b090c6f0c6aa6307ab60c64ecf1c9604da9056d7dcc279ec70" Nov 04 01:30:01.949049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.946077 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf" Nov 04 01:30:01.950569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.950542 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerStarted","Data":"1bc2533e528414a84fed264ffe39409ee1c69a31cf27c1c629ead901a67df034"} Nov 04 01:30:01.952436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.952411 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerStarted","Data":"ff2ffaf059b52c3faf33e6bce50f3694cc98d1310697c0fb58d8822df3c3a316"} Nov 04 01:30:01.954538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.954504 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerStarted","Data":"d24b0eb6898f727d5f93cd4523b19ef8dea5f20c25f9707ee02971e47b9ce202"} Nov 04 01:30:01.954640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.954555 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerStarted","Data":"ad4661eb9614f6ff69c0088e1200079813ff5974d51b9af6499c019bb607ea5e"} Nov 04 01:30:01.956712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.956569 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerStarted","Data":"20091b96accbe53b69eb80efd968ba8039c8b1776df90d1349b0f949f9170edd"} Nov 04 01:30:01.958663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.958639 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerStarted","Data":"890e7a61986b46e8ecf65211a9d432294c35b2cdf3a081c15608e1016e37a6e4"} Nov 04 01:30:01.960899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:01.960386 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerStarted","Data":"cf495ed5f1b5ec9e174eb80d87a566fb3ef1805b145051704614644feb046329"} Nov 04 01:30:02.051167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.051110 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" podStartSLOduration=2.051092924 podStartE2EDuration="2.051092924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.050591822 +0000 UTC m=+1970.809253983" watchObservedRunningTime="2026-11-04 01:30:02.051092924 +0000 UTC m=+1970.809755087" Nov 04 01:30:02.161716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.161659 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" podStartSLOduration=2.16164112 podStartE2EDuration="2.16164112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.160746279 +0000 UTC m=+1970.919408443" watchObservedRunningTime="2026-11-04 01:30:02.16164112 +0000 UTC m=+1970.920303357" Nov 04 01:30:02.273458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.273407 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" podStartSLOduration=2.2733923750000002 podStartE2EDuration="2.273392375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.269381712 +0000 UTC m=+1971.028043872" watchObservedRunningTime="2026-11-04 01:30:02.273392375 +0000 UTC m=+1971.032054573" Nov 04 01:30:02.283401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.283370 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:02.283518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.283422 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:02.431080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.430664 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" podStartSLOduration=2.430645764 podStartE2EDuration="2.430645764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.428820968 +0000 UTC m=+1971.187483141" watchObservedRunningTime="2026-11-04 01:30:02.430645764 +0000 UTC m=+1971.189307928" Nov 04 01:30:02.545793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.545740 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" podStartSLOduration=2.545724703 podStartE2EDuration="2.545724703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:02.539752612 +0000 UTC m=+1971.298414773" watchObservedRunningTime="2026-11-04 01:30:02.545724703 +0000 UTC m=+1971.304386870" Nov 04 01:30:02.797351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.797310 4486 patch_prober.go:29] interesting pod/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:02.797557 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.797380 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:02.973564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.973473 4486 generic.go:356] "Generic (PLEG): container finished" podID="343e4987-1f43-45b6-b654-497177789217" containerID="969673e8f9782be19cbaf1918ca07d24148dc60b284d32271ea682a75b375889" exitCode=60 Nov 04 01:30:02.973757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.973558 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerDied","Data":"969673e8f9782be19cbaf1918ca07d24148dc60b284d32271ea682a75b375889"} Nov 04 01:30:02.973904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.973884 4486 scope.go:122] "RemoveContainer" containerID="969673e8f9782be19cbaf1918ca07d24148dc60b284d32271ea682a75b375889" Nov 04 01:30:02.976103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.976076 4486 generic.go:356] "Generic (PLEG): container finished" podID="21ccff16-040a-400e-97b2-be728905afa5" containerID="626fb310d75f15cbd3368ba1e9cf2c3a529c4b724839fcbffc7fa237cf42071f" exitCode=60 Nov 04 01:30:02.976226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.976107 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerDied","Data":"626fb310d75f15cbd3368ba1e9cf2c3a529c4b724839fcbffc7fa237cf42071f"} Nov 04 01:30:02.976439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.976422 4486 scope.go:122] "RemoveContainer" containerID="626fb310d75f15cbd3368ba1e9cf2c3a529c4b724839fcbffc7fa237cf42071f" Nov 04 01:30:02.978269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.978248 4486 generic.go:356] "Generic (PLEG): container finished" podID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerID="d7db44c04333b85019c4438f264822d8c88c8f90e049b4ef5f452ee9c713a970" exitCode=7 Nov 04 01:30:02.978351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.978282 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerDied","Data":"d7db44c04333b85019c4438f264822d8c88c8f90e049b4ef5f452ee9c713a970"} Nov 04 01:30:02.978614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.978594 4486 scope.go:122] "RemoveContainer" containerID="d7db44c04333b85019c4438f264822d8c88c8f90e049b4ef5f452ee9c713a970" Nov 04 01:30:02.980402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.980380 4486 generic.go:356] "Generic (PLEG): container finished" podID="445d79ac-a030-483b-a248-926b0107bfb9" containerID="7bd1ae4b9b7c756e51c2bef1a52a62f86c282bda8227a45292db620104e53f5e" exitCode=60 Nov 04 01:30:02.980513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.980474 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerDied","Data":"7bd1ae4b9b7c756e51c2bef1a52a62f86c282bda8227a45292db620104e53f5e"} Nov 04 01:30:02.980795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.980778 4486 scope.go:122] "RemoveContainer" containerID="7bd1ae4b9b7c756e51c2bef1a52a62f86c282bda8227a45292db620104e53f5e" Nov 04 01:30:02.982658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.982584 4486 generic.go:356] "Generic (PLEG): container finished" podID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerID="47eb698e5b42f4c83af0f2be7df6abc0ab92febf064d1a6d592e921085e037b5" exitCode=60 Nov 04 01:30:02.982658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.982609 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerDied","Data":"47eb698e5b42f4c83af0f2be7df6abc0ab92febf064d1a6d592e921085e037b5"} Nov 04 01:30:02.982897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.982881 4486 scope.go:122] "RemoveContainer" containerID="47eb698e5b42f4c83af0f2be7df6abc0ab92febf064d1a6d592e921085e037b5" Nov 04 01:30:02.984716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.984696 4486 generic.go:356] "Generic (PLEG): container finished" podID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerID="65b5a604eea3616289f7f02c3562904fd92bb7100eaba11a7c150351943c0a6e" exitCode=60 Nov 04 01:30:02.984843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.984826 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerDied","Data":"65b5a604eea3616289f7f02c3562904fd92bb7100eaba11a7c150351943c0a6e"} Nov 04 01:30:02.985265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.985247 4486 scope.go:122] "RemoveContainer" containerID="65b5a604eea3616289f7f02c3562904fd92bb7100eaba11a7c150351943c0a6e" Nov 04 01:30:02.990418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.990393 4486 generic.go:356] "Generic (PLEG): container finished" podID="888c5373-4a1d-4f15-8769-128f60a0241d" containerID="9d75d086f14576916b62cc607c4be72bcf1c67726822c18d24dc0a229201eff0" exitCode=60 Nov 04 01:30:02.990529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.990488 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerDied","Data":"9d75d086f14576916b62cc607c4be72bcf1c67726822c18d24dc0a229201eff0"} Nov 04 01:30:02.990859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.990685 4486 scope.go:122] "RemoveContainer" containerID="9d75d086f14576916b62cc607c4be72bcf1c67726822c18d24dc0a229201eff0" Nov 04 01:30:02.992451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.992432 4486 generic.go:356] "Generic (PLEG): container finished" podID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerID="d24b0eb6898f727d5f93cd4523b19ef8dea5f20c25f9707ee02971e47b9ce202" exitCode=60 Nov 04 01:30:02.992578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.992509 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerDied","Data":"d24b0eb6898f727d5f93cd4523b19ef8dea5f20c25f9707ee02971e47b9ce202"} Nov 04 01:30:02.992727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:02.992711 4486 scope.go:122] "RemoveContainer" containerID="d24b0eb6898f727d5f93cd4523b19ef8dea5f20c25f9707ee02971e47b9ce202" Nov 04 01:30:04.003039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.002972 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerStarted","Data":"6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791"} Nov 04 01:30:04.005340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.005311 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerStarted","Data":"1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1"} Nov 04 01:30:04.007345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.007318 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerStarted","Data":"95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473"} Nov 04 01:30:04.009317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.009294 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerStarted","Data":"23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708"} Nov 04 01:30:04.011155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.011124 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerStarted","Data":"b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16"} Nov 04 01:30:04.013030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.013008 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerStarted","Data":"def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207"} Nov 04 01:30:04.015035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.015011 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerStarted","Data":"a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1"} Nov 04 01:30:04.016864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.016839 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerStarted","Data":"a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a"} Nov 04 01:30:04.121322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.121153 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" podStartSLOduration=4.121134779 podStartE2EDuration="4.121134779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:04.118306774 +0000 UTC m=+1972.876968936" watchObservedRunningTime="2026-11-04 01:30:04.121134779 +0000 UTC m=+1972.879796944" Nov 04 01:30:04.361171 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.361117 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" podStartSLOduration=4.361102385 podStartE2EDuration="4.361102385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:04.35495116 +0000 UTC m=+1973.113613322" watchObservedRunningTime="2026-11-04 01:30:04.361102385 +0000 UTC m=+1973.119764546" Nov 04 01:30:04.695973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.695921 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" podStartSLOduration=4.695906424 podStartE2EDuration="4.695906424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:04.6942803 +0000 UTC m=+1973.452942465" watchObservedRunningTime="2026-11-04 01:30:04.695906424 +0000 UTC m=+1973.454568581" Nov 04 01:30:04.761573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.761537 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:30:04.766925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.766901 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:30:04.871218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871186 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871256 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871306 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871347 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871391 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871417 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871468 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871490 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871510 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871508 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871528 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871561 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871612 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871650 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871679 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.871730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871695 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.872374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871742 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.872374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.871755 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.872374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.872002 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.872542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.872511 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\") pod \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\" (UID: \"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25\") " Nov 04 01:30:04.872609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.872556 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\") pod \"d09a2489-14fb-408f-9842-1736efb0e3d4\" (UID: \"d09a2489-14fb-408f-9842-1736efb0e3d4\") " Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873130 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873338 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873503 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873539 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873561 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873576 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873591 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873605 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.873886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873655 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.874536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.873938 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.874939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.874913 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.875061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.875037 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "kube-api-access-cz9xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.875342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.875315 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.875515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.875492 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.876560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.876533 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "kube-api-access-c6r2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.876651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.876611 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.877218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.877191 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.878193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.878171 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo" pod "d09a2489-14fb-408f-9842-1736efb0e3d4" (UID: "d09a2489-14fb-408f-9842-1736efb0e3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.878463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.878443 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.879628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.879595 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo" pod "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" (UID: "3be6c0dd-2e33-4dc0-8e81-e0684a23fd25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.900619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.900560 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:30:04.974103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974068 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974138 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974171 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974239 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974275 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974330 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974384 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974399 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.974599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974502 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974537 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\") pod \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\" (UID: \"7d05d797-e34e-4cb3-959c-c6e3f75235b9\") " Nov 04 01:30:04.974753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974727 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:04.974753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.974732 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.975054 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975032 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.975132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975069 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:04.975312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975283 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975309 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975327 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9xg\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-kube-api-access-cz9xg\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975343 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975360 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975375 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975389 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975403 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975416 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975429 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d09a2489-14fb-408f-9842-1736efb0e3d4-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975444 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d09a2489-14fb-408f-9842-1736efb0e3d4-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975457 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975471 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975485 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975498 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d09a2489-14fb-408f-9842-1736efb0e3d4-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975514 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6r2k\" (UniqueName: \"kubernetes.io/projected/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-kube-api-access-c6r2k\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975528 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.975717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.975544 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:04.976953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.976925 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.976953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.976943 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "kube-api-access-8rcrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:04.976953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.976925 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:04.977162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:04.977045 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy" pod "7d05d797-e34e-4cb3-959c-c6e3f75235b9" (UID: "7d05d797-e34e-4cb3-959c-c6e3f75235b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:05.027178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.027143 4486 generic.go:356] "Generic (PLEG): container finished" podID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerID="d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c" exitCode=0 Nov 04 01:30:05.027585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.027222 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" Nov 04 01:30:05.027585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.027235 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" event={"ID":"7d05d797-e34e-4cb3-959c-c6e3f75235b9","Type":"ContainerDied","Data":"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c"} Nov 04 01:30:05.027585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.027317 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb" event={"ID":"7d05d797-e34e-4cb3-959c-c6e3f75235b9","Type":"ContainerDied","Data":"7c0a10093eb0e8fa21cae76c5ef8b933aab5db25517e2bd5169ed3c545e056f9"} Nov 04 01:30:05.027585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.027348 4486 scope.go:122] "RemoveContainer" containerID="d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c" Nov 04 01:30:05.029575 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.029550 4486 generic.go:356] "Generic (PLEG): container finished" podID="343e4987-1f43-45b6-b654-497177789217" containerID="23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708" exitCode=60 Nov 04 01:30:05.029909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.029883 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerDied","Data":"23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708"} Nov 04 01:30:05.030036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.029917 4486 scope.go:122] "RemoveContainer" containerID="23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708" Nov 04 01:30:05.030292 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.030267 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-v6pp8_odh-ai-gateway-infra(343e4987-1f43-45b6-b654-497177789217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" podUID="343e4987-1f43-45b6-b654-497177789217" Nov 04 01:30:05.031775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.031742 4486 generic.go:356] "Generic (PLEG): container finished" podID="21ccff16-040a-400e-97b2-be728905afa5" containerID="b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16" exitCode=60 Nov 04 01:30:05.031879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.031834 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerDied","Data":"b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16"} Nov 04 01:30:05.032301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.032284 4486 scope.go:122] "RemoveContainer" containerID="b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16" Nov 04 01:30:05.032545 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.032525 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8_odh-ai-gateway-infra(21ccff16-040a-400e-97b2-be728905afa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" podUID="21ccff16-040a-400e-97b2-be728905afa5" Nov 04 01:30:05.034917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.034895 4486 generic.go:356] "Generic (PLEG): container finished" podID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerID="def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207" exitCode=7 Nov 04 01:30:05.035072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.034996 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerDied","Data":"def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207"} Nov 04 01:30:05.035230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.035209 4486 scope.go:122] "RemoveContainer" containerID="def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207" Nov 04 01:30:05.035485 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.035465 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq_odh-ai-gateway-infra(9695423d-854a-4fac-a1f8-7185c8eb4a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" Nov 04 01:30:05.037546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.037523 4486 generic.go:356] "Generic (PLEG): container finished" podID="445d79ac-a030-483b-a248-926b0107bfb9" containerID="a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1" exitCode=60 Nov 04 01:30:05.037619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.037600 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerDied","Data":"a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1"} Nov 04 01:30:05.037819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.037800 4486 scope.go:122] "RemoveContainer" containerID="a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1" Nov 04 01:30:05.038135 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.038112 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x_odh-ai-gateway-infra(445d79ac-a030-483b-a248-926b0107bfb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" podUID="445d79ac-a030-483b-a248-926b0107bfb9" Nov 04 01:30:05.039685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.039665 4486 generic.go:356] "Generic (PLEG): container finished" podID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerID="a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a" exitCode=60 Nov 04 01:30:05.039780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.039741 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerDied","Data":"a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a"} Nov 04 01:30:05.039956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.039941 4486 scope.go:122] "RemoveContainer" containerID="a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a" Nov 04 01:30:05.040212 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.040193 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh_odh-ai-gateway-infra(efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" Nov 04 01:30:05.040721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.040707 4486 scope.go:122] "RemoveContainer" containerID="d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c" Nov 04 01:30:05.040965 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.040942 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c\": container with ID starting with d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c not found: ID does not exist" containerID="d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c" Nov 04 01:30:05.041063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.040973 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c"} err="failed to get container status \"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c\": rpc error: code = NotFound desc = could not find container \"d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c\": container with ID starting with d30be1277d66d5c6f49c1a48ec638c6b5cd7f8e18fc74b836ebbcc3aa91e541c not found: ID does not exist" Nov 04 01:30:05.041063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.041020 4486 scope.go:122] "RemoveContainer" containerID="969673e8f9782be19cbaf1918ca07d24148dc60b284d32271ea682a75b375889" Nov 04 01:30:05.041451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.041435 4486 generic.go:356] "Generic (PLEG): container finished" podID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerID="69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1" exitCode=0 Nov 04 01:30:05.041512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.041459 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerDied","Data":"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1"} Nov 04 01:30:05.041512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.041490 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" Nov 04 01:30:05.041512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.041505 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps" event={"ID":"3be6c0dd-2e33-4dc0-8e81-e0684a23fd25","Type":"ContainerDied","Data":"037c99186b47e366bf88656c91d7e3a7163300f308454732ba38a05b70fe81b8"} Nov 04 01:30:05.044189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.044170 4486 generic.go:356] "Generic (PLEG): container finished" podID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerID="6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791" exitCode=60 Nov 04 01:30:05.044290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.044197 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerDied","Data":"6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791"} Nov 04 01:30:05.044456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.044441 4486 scope.go:122] "RemoveContainer" containerID="6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791" Nov 04 01:30:05.044690 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.044665 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts_odh-ai-gateway-infra(766b3721-ffd9-4f93-8cf2-172ccd39887d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" Nov 04 01:30:05.046517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.046492 4486 generic.go:356] "Generic (PLEG): container finished" podID="888c5373-4a1d-4f15-8769-128f60a0241d" containerID="1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1" exitCode=60 Nov 04 01:30:05.046608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.046579 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerDied","Data":"1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1"} Nov 04 01:30:05.046939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.046888 4486 scope.go:122] "RemoveContainer" containerID="1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1" Nov 04 01:30:05.047214 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.047157 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6_odh-ai-gateway-infra(888c5373-4a1d-4f15-8769-128f60a0241d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" Nov 04 01:30:05.050393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.049678 4486 generic.go:356] "Generic (PLEG): container finished" podID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerID="95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473" exitCode=60 Nov 04 01:30:05.050393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.049776 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerDied","Data":"95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473"} Nov 04 01:30:05.050393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.050109 4486 scope.go:122] "RemoveContainer" containerID="95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473" Nov 04 01:30:05.050393 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.050332 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs_odh-ai-gateway-infra(aac273c8-2b61-4570-9a34-eb572bfcedcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" Nov 04 01:30:05.053899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.052387 4486 generic.go:356] "Generic (PLEG): container finished" podID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerID="b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995" exitCode=0 Nov 04 01:30:05.053899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.052426 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" event={"ID":"d09a2489-14fb-408f-9842-1736efb0e3d4","Type":"ContainerDied","Data":"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995"} Nov 04 01:30:05.053899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.052452 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" event={"ID":"d09a2489-14fb-408f-9842-1736efb0e3d4","Type":"ContainerDied","Data":"26fec395c3ceba29079afd6bda1d8c26c75d61c5af96687e7acf2ba40ace0520"} Nov 04 01:30:05.053899 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.052498 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg" Nov 04 01:30:05.061683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.061661 4486 scope.go:122] "RemoveContainer" containerID="626fb310d75f15cbd3368ba1e9cf2c3a529c4b724839fcbffc7fa237cf42071f" Nov 04 01:30:05.077533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.077503 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rcrc\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-kube-api-access-8rcrc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:05.077720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.077568 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:05.077720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.077589 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:05.077720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.077607 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d05d797-e34e-4cb3-959c-c6e3f75235b9-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:05.080313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.080290 4486 scope.go:122] "RemoveContainer" containerID="d7db44c04333b85019c4438f264822d8c88c8f90e049b4ef5f452ee9c713a970" Nov 04 01:30:05.097469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.097447 4486 scope.go:122] "RemoveContainer" containerID="7bd1ae4b9b7c756e51c2bef1a52a62f86c282bda8227a45292db620104e53f5e" Nov 04 01:30:05.113147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.113127 4486 scope.go:122] "RemoveContainer" containerID="47eb698e5b42f4c83af0f2be7df6abc0ab92febf064d1a6d592e921085e037b5" Nov 04 01:30:05.127152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.127128 4486 scope.go:122] "RemoveContainer" containerID="69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1" Nov 04 01:30:05.161027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.160999 4486 scope.go:122] "RemoveContainer" containerID="fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1" Nov 04 01:30:05.173951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.173921 4486 scope.go:122] "RemoveContainer" containerID="69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1" Nov 04 01:30:05.189041 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.175451 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1\": container with ID starting with 69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1 not found: ID does not exist" containerID="69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1" Nov 04 01:30:05.189041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.175501 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1"} err="failed to get container status \"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1\": rpc error: code = NotFound desc = could not find container \"69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1\": container with ID starting with 69f2afab97cecff73eb652200e8b2ae700ca0dd7f4058a2100fa90e904ca4aa1 not found: ID does not exist" Nov 04 01:30:05.189041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.175530 4486 scope.go:122] "RemoveContainer" containerID="fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1" Nov 04 01:30:05.204001 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.189825 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1\": container with ID starting with fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1 not found: ID does not exist" containerID="fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1" Nov 04 01:30:05.204001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.189871 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1"} err="failed to get container status \"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1\": rpc error: code = NotFound desc = could not find container \"fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1\": container with ID starting with fb99474a27bc142d4df821b2145af07dc5c71069033a6c0b9148147c509610e1 not found: ID does not exist" Nov 04 01:30:05.204001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.189896 4486 scope.go:122] "RemoveContainer" containerID="65b5a604eea3616289f7f02c3562904fd92bb7100eaba11a7c150351943c0a6e" Nov 04 01:30:05.231057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.231027 4486 scope.go:122] "RemoveContainer" containerID="9d75d086f14576916b62cc607c4be72bcf1c67726822c18d24dc0a229201eff0" Nov 04 01:30:05.245291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.245267 4486 scope.go:122] "RemoveContainer" containerID="d24b0eb6898f727d5f93cd4523b19ef8dea5f20c25f9707ee02971e47b9ce202" Nov 04 01:30:05.258949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.258928 4486 scope.go:122] "RemoveContainer" containerID="b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995" Nov 04 01:30:05.292866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.292837 4486 scope.go:122] "RemoveContainer" containerID="b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995" Nov 04 01:30:05.293253 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:05.293226 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995\": container with ID starting with b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995 not found: ID does not exist" containerID="b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995" Nov 04 01:30:05.293370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.293262 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995"} err="failed to get container status \"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995\": rpc error: code = NotFound desc = could not find container \"b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995\": container with ID starting with b24d253ee6b2e4236f64a23edf5933c64a1b1712d5a7edfd880b4aa68d7c5995 not found: ID does not exist" Nov 04 01:30:05.401824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.401787 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:30:05.412094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.412003 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-9bttb"] Nov 04 01:30:05.516763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.516727 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:30:05.521189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:05.521165 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-qq9ps"] Nov 04 01:30:06.068848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.068818 4486 scope.go:122] "RemoveContainer" containerID="a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a" Nov 04 01:30:06.069379 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.069258 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh_odh-ai-gateway-infra(efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" Nov 04 01:30:06.071817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.071794 4486 scope.go:122] "RemoveContainer" containerID="6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791" Nov 04 01:30:06.072112 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.072090 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts_odh-ai-gateway-infra(766b3721-ffd9-4f93-8cf2-172ccd39887d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" Nov 04 01:30:06.073389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.073366 4486 scope.go:122] "RemoveContainer" containerID="1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1" Nov 04 01:30:06.073639 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.073609 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6_odh-ai-gateway-infra(888c5373-4a1d-4f15-8769-128f60a0241d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" Nov 04 01:30:06.074862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.074841 4486 scope.go:122] "RemoveContainer" containerID="95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473" Nov 04 01:30:06.075125 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.075104 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs_odh-ai-gateway-infra(aac273c8-2b61-4570-9a34-eb572bfcedcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" Nov 04 01:30:06.078181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.078163 4486 scope.go:122] "RemoveContainer" containerID="23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708" Nov 04 01:30:06.078394 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.078372 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-v6pp8_odh-ai-gateway-infra(343e4987-1f43-45b6-b654-497177789217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" podUID="343e4987-1f43-45b6-b654-497177789217" Nov 04 01:30:06.079769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.079753 4486 scope.go:122] "RemoveContainer" containerID="b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16" Nov 04 01:30:06.080023 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.079996 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8_odh-ai-gateway-infra(21ccff16-040a-400e-97b2-be728905afa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" podUID="21ccff16-040a-400e-97b2-be728905afa5" Nov 04 01:30:06.081747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.081727 4486 scope.go:122] "RemoveContainer" containerID="def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207" Nov 04 01:30:06.081958 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.081938 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq_odh-ai-gateway-infra(9695423d-854a-4fac-a1f8-7185c8eb4a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" Nov 04 01:30:06.083377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.083359 4486 scope.go:122] "RemoveContainer" containerID="a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1" Nov 04 01:30:06.083587 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:06.083568 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x_odh-ai-gateway-infra(445d79ac-a030-483b-a248-926b0107bfb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" podUID="445d79ac-a030-483b-a248-926b0107bfb9" Nov 04 01:30:06.444467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.444378 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:30:06.464433 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:06.464390 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c563ee-openshift-default-7588b46465-gdvpg"] Nov 04 01:30:07.177962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.177911 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52"] Nov 04 01:30:07.178744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178721 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" Nov 04 01:30:07.178816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178746 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" Nov 04 01:30:07.178816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178787 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.178816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178796 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.178816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178810 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.178961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178818 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.178961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178835 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" Nov 04 01:30:07.178961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178843 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" Nov 04 01:30:07.178961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178878 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" containerName="revoke-keys" Nov 04 01:30:07.178961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.178884 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" containerName="revoke-keys" Nov 04 01:30:07.179183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.179027 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.179183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.179038 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" containerName="istio-proxy" Nov 04 01:30:07.179183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.179045 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" containerName="istio-proxy" Nov 04 01:30:07.179183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.179052 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" containerName="revoke-keys" Nov 04 01:30:07.179183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.179063 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" containerName="istio-proxy" Nov 04 01:30:07.183949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.183930 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.248096 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.241379 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52"] Nov 04 01:30:07.312100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.312053 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.312288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.312251 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9gn\" (UniqueName: \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.312417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.312401 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.318897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.318874 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:30:07.354494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.354448 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be6c0dd-2e33-4dc0-8e81-e0684a23fd25" path="/var/lib/kubelet/pods/3be6c0dd-2e33-4dc0-8e81-e0684a23fd25/volumes" Nov 04 01:30:07.355280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.355255 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d05d797-e34e-4cb3-959c-c6e3f75235b9" path="/var/lib/kubelet/pods/7d05d797-e34e-4cb3-959c-c6e3f75235b9/volumes" Nov 04 01:30:07.355916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.355896 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09a2489-14fb-408f-9842-1736efb0e3d4" path="/var/lib/kubelet/pods/d09a2489-14fb-408f-9842-1736efb0e3d4/volumes" Nov 04 01:30:07.413482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.413452 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.413639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.413610 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.414115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.413959 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9gn\" (UniqueName: \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.414259 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.414225 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.416295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.416266 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.468922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.468837 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9gn\" (UniqueName: \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.498762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.498725 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:07.881061 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:07.881026 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21deeee9_3177_4639_866e_f49e78b5593b.slice/crio-7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0 WatchSource:0}: Error finding container 7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0: Status 404 returned error can't find the container with id 7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0 Nov 04 01:30:07.881240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:07.881221 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52"] Nov 04 01:30:08.103047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:08.103009 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" event={"ID":"21deeee9-3177-4639-866e-f49e78b5593b","Type":"ContainerStarted","Data":"7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0"} Nov 04 01:30:09.112885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:09.112849 4486 generic.go:356] "Generic (PLEG): container finished" podID="21deeee9-3177-4639-866e-f49e78b5593b" containerID="e8a4b87b0d2e1110ab9542c7130a4027b32eabba091150db2999a8fa02bff608" exitCode=0 Nov 04 01:30:09.113362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:09.112934 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" event={"ID":"21deeee9-3177-4639-866e-f49e78b5593b","Type":"ContainerDied","Data":"e8a4b87b0d2e1110ab9542c7130a4027b32eabba091150db2999a8fa02bff608"} Nov 04 01:30:10.295153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.295124 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:10.361838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.361803 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\" (UniqueName: \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\") pod \"21deeee9-3177-4639-866e-f49e78b5593b\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " Nov 04 01:30:10.362008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.361952 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\") pod \"21deeee9-3177-4639-866e-f49e78b5593b\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " Nov 04 01:30:10.362077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.362052 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\") pod \"21deeee9-3177-4639-866e-f49e78b5593b\" (UID: \"21deeee9-3177-4639-866e-f49e78b5593b\") " Nov 04 01:30:10.362450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.362404 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca" pod "21deeee9-3177-4639-866e-f49e78b5593b" (UID: "21deeee9-3177-4639-866e-f49e78b5593b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:10.362656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.362634 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:10.364190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.364162 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn" pod "21deeee9-3177-4639-866e-f49e78b5593b" (UID: "21deeee9-3177-4639-866e-f49e78b5593b"). InnerVolumeSpecName "kube-api-access-tk9gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:10.364286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.364208 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca" pod "21deeee9-3177-4639-866e-f49e78b5593b" (UID: "21deeee9-3177-4639-866e-f49e78b5593b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:10.465029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.464568 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9gn\" (UniqueName: \"kubernetes.io/projected/21deeee9-3177-4639-866e-f49e78b5593b-kube-api-access-tk9gn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:10.465029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:10.464606 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21deeee9-3177-4639-866e-f49e78b5593b-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:11.131320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:11.131281 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" event={"ID":"21deeee9-3177-4639-866e-f49e78b5593b","Type":"ContainerDied","Data":"7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0"} Nov 04 01:30:11.131482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:11.131350 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7727e43b7ef89e1a1fd161a7c37dc21d4638b21c596744b10b82ba9fa400d4e0" Nov 04 01:30:11.131482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:11.131298 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52" Nov 04 01:30:13.912027 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:13.911058 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:30:15.342455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:15.342421 4486 scope.go:122] "RemoveContainer" containerID="a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a" Nov 04 01:30:15.342892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:15.342489 4486 scope.go:122] "RemoveContainer" containerID="23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708" Nov 04 01:30:16.180147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:16.180108 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerStarted","Data":"e7f892cfcbb3fe708ade891601d0dbc033c9d384cc078ffa81a98a7c9cffb6e2"} Nov 04 01:30:16.182201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:16.182172 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerStarted","Data":"563a1ed8c378d6781dfb380de1a0f41e1f06a7db4b64032c09def0f004966a79"} Nov 04 01:30:17.192580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.192539 4486 generic.go:356] "Generic (PLEG): container finished" podID="343e4987-1f43-45b6-b654-497177789217" containerID="e7f892cfcbb3fe708ade891601d0dbc033c9d384cc078ffa81a98a7c9cffb6e2" exitCode=60 Nov 04 01:30:17.193070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.192614 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerDied","Data":"e7f892cfcbb3fe708ade891601d0dbc033c9d384cc078ffa81a98a7c9cffb6e2"} Nov 04 01:30:17.193070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.192685 4486 scope.go:122] "RemoveContainer" containerID="23f5eb06d7efdf7cfb18f4cdee40c975145acea07f0143fbcb7b652f985f8708" Nov 04 01:30:17.193178 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.193098 4486 scope.go:122] "RemoveContainer" containerID="e7f892cfcbb3fe708ade891601d0dbc033c9d384cc078ffa81a98a7c9cffb6e2" Nov 04 01:30:17.193435 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:17.193417 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-v6pp8_odh-ai-gateway-infra(343e4987-1f43-45b6-b654-497177789217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" podUID="343e4987-1f43-45b6-b654-497177789217" Nov 04 01:30:17.195486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.195466 4486 generic.go:356] "Generic (PLEG): container finished" podID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerID="563a1ed8c378d6781dfb380de1a0f41e1f06a7db4b64032c09def0f004966a79" exitCode=60 Nov 04 01:30:17.195572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.195511 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerDied","Data":"563a1ed8c378d6781dfb380de1a0f41e1f06a7db4b64032c09def0f004966a79"} Nov 04 01:30:17.195880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.195866 4486 scope.go:122] "RemoveContainer" containerID="563a1ed8c378d6781dfb380de1a0f41e1f06a7db4b64032c09def0f004966a79" Nov 04 01:30:17.196175 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:17.196159 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh_odh-ai-gateway-infra(efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" Nov 04 01:30:17.206779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.206761 4486 scope.go:122] "RemoveContainer" containerID="a363220bb4a42b87107ca05bc9ea61b1a0bf2312d796b02b8d918feacdabe40a" Nov 04 01:30:17.313494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.313464 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:17.362698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.362658 4486 scope.go:122] "RemoveContainer" containerID="a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1" Nov 04 01:30:17.362905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.362878 4486 scope.go:122] "RemoveContainer" containerID="def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207" Nov 04 01:30:17.363228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.363205 4486 scope.go:122] "RemoveContainer" containerID="95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473" Nov 04 01:30:17.363325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.363301 4486 scope.go:122] "RemoveContainer" containerID="6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791" Nov 04 01:30:17.363413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.363397 4486 scope.go:122] "RemoveContainer" containerID="b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16" Nov 04 01:30:17.475220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.475143 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:17.811005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.809381 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:30:17.854003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.848041 4486 cpu_manager.go:414] "containerMap: removing container" podUID="21deeee9-3177-4639-866e-f49e78b5593b" containerName="revoke-keys" Nov 04 01:30:17.854003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.848098 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21deeee9-3177-4639-866e-f49e78b5593b" containerName="revoke-keys" Nov 04 01:30:17.854003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.848475 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="21deeee9-3177-4639-866e-f49e78b5593b" containerName="revoke-keys" Nov 04 01:30:17.896005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.889007 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.969486 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-fc46df-openshift-default-dockercfg-cnzkk\"" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.969827 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.969873 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.969924 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.969973 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.970112 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.970252 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.970304 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.970351 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.006025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:17.970378 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hhn\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082324 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082432 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082468 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082498 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082519 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hhn\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082566 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082587 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082620 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.082653 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.083388 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.084084 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.084567 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.089228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.084808 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.093528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.091350 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.107749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.107671 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.108392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.108108 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:30:18.108798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.108755 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.154087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.154051 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hhn\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.160002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.156667 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\") pod \"e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.205496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.205455 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerStarted","Data":"ec43a70944182292350223314321ba61865c7268f0d27cb84b30b1c2b88d5881"} Nov 04 01:30:18.210536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.210276 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerStarted","Data":"3e9f13b73a50c7856cf9df39d98fb8b5a77a1c5f151d835226778639f0353f40"} Nov 04 01:30:18.212725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.212697 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerStarted","Data":"709b2cb6d9ee6819eaa148722d40f503e646950685af0c221680a81f2ab12b43"} Nov 04 01:30:18.214948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.214469 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerStarted","Data":"288c85865aae34200b089c1b6e7a8dc8753f35349dbbcacd3d9f1338b0e9567b"} Nov 04 01:30:18.218345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.218322 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerStarted","Data":"667b882066b1ba9f9dbaef76e6ce136a8aba15e56ecef8f3cd36b199c02fe888"} Nov 04 01:30:18.364203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.363209 4486 scope.go:122] "RemoveContainer" containerID="1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1" Nov 04 01:30:18.406322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.406290 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:18.484940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.484870 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:18.590539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.590395 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\" (UniqueName: \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\") pod \"343e4987-1f43-45b6-b654-497177789217\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " Nov 04 01:30:18.590709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.590549 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\") pod \"343e4987-1f43-45b6-b654-497177789217\" (UID: \"343e4987-1f43-45b6-b654-497177789217\") " Nov 04 01:30:18.600295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.600121 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca" pod "343e4987-1f43-45b6-b654-497177789217" (UID: "343e4987-1f43-45b6-b654-497177789217"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:18.604095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.602591 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9" pod "343e4987-1f43-45b6-b654-497177789217" (UID: "343e4987-1f43-45b6-b654-497177789217"). InnerVolumeSpecName "kube-api-access-mgrj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:18.627956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.627933 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:18.691746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.691709 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\" (UniqueName: \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\") pod \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " Nov 04 01:30:18.691954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.691825 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\") pod \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\" (UID: \"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff\") " Nov 04 01:30:18.692642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.692592 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/343e4987-1f43-45b6-b654-497177789217-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.692642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.692624 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrj9\" (UniqueName: \"kubernetes.io/projected/343e4987-1f43-45b6-b654-497177789217-kube-api-access-mgrj9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.692797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.692712 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca" pod "efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" (UID: "efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:18.694671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.694632 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7" pod "efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" (UID: "efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff"). InnerVolumeSpecName "kube-api-access-lmdj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:18.793747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.793720 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdj7\" (UniqueName: \"kubernetes.io/projected/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-kube-api-access-lmdj7\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.793903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.793752 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:18.904707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.904681 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:30:18.958288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.958089 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:18.958288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.958187 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:18.958288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:18.958234 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:19.292418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.292343 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" Nov 04 01:30:19.293343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.292427 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh" event={"ID":"efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff","Type":"ContainerDied","Data":"65a6594214abe0337a7a022d56f168b5029d6806b06bdaaeca7c44489c1b3423"} Nov 04 01:30:19.293481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.293382 4486 scope.go:122] "RemoveContainer" containerID="563a1ed8c378d6781dfb380de1a0f41e1f06a7db4b64032c09def0f004966a79" Nov 04 01:30:19.298046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.297342 4486 generic.go:356] "Generic (PLEG): container finished" podID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerID="667b882066b1ba9f9dbaef76e6ce136a8aba15e56ecef8f3cd36b199c02fe888" exitCode=60 Nov 04 01:30:19.298046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.297463 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerDied","Data":"667b882066b1ba9f9dbaef76e6ce136a8aba15e56ecef8f3cd36b199c02fe888"} Nov 04 01:30:19.298046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.297847 4486 scope.go:122] "RemoveContainer" containerID="667b882066b1ba9f9dbaef76e6ce136a8aba15e56ecef8f3cd36b199c02fe888" Nov 04 01:30:19.298444 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:19.298422 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts_odh-ai-gateway-infra(766b3721-ffd9-4f93-8cf2-172ccd39887d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" Nov 04 01:30:19.302608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.302588 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerStarted","Data":"e36aa4e0cfe88eaed481514c5db7b34609edeb2ca17f2b066b97c8fd8c0243ab"} Nov 04 01:30:19.305143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.305122 4486 generic.go:356] "Generic (PLEG): container finished" podID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerID="ec43a70944182292350223314321ba61865c7268f0d27cb84b30b1c2b88d5881" exitCode=60 Nov 04 01:30:19.308717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.305370 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerDied","Data":"ec43a70944182292350223314321ba61865c7268f0d27cb84b30b1c2b88d5881"} Nov 04 01:30:19.308717 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.307539 4486 scope.go:122] "RemoveContainer" containerID="ec43a70944182292350223314321ba61865c7268f0d27cb84b30b1c2b88d5881" Nov 04 01:30:19.308717 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:19.307786 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs_odh-ai-gateway-infra(aac273c8-2b61-4570-9a34-eb572bfcedcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" Nov 04 01:30:19.309632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.309606 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" event={"ID":"343e4987-1f43-45b6-b654-497177789217","Type":"ContainerDied","Data":"20091b96accbe53b69eb80efd968ba8039c8b1776df90d1349b0f949f9170edd"} Nov 04 01:30:19.309716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.309701 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8" Nov 04 01:30:19.316449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.316421 4486 generic.go:356] "Generic (PLEG): container finished" podID="21ccff16-040a-400e-97b2-be728905afa5" containerID="3e9f13b73a50c7856cf9df39d98fb8b5a77a1c5f151d835226778639f0353f40" exitCode=60 Nov 04 01:30:19.316569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.316507 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerDied","Data":"3e9f13b73a50c7856cf9df39d98fb8b5a77a1c5f151d835226778639f0353f40"} Nov 04 01:30:19.316824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.316803 4486 scope.go:122] "RemoveContainer" containerID="3e9f13b73a50c7856cf9df39d98fb8b5a77a1c5f151d835226778639f0353f40" Nov 04 01:30:19.317081 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:19.317057 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8_odh-ai-gateway-infra(21ccff16-040a-400e-97b2-be728905afa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" podUID="21ccff16-040a-400e-97b2-be728905afa5" Nov 04 01:30:19.418070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.418041 4486 scope.go:122] "RemoveContainer" containerID="6b1dd10d53bc679477f7b5e84a0b078c1faa0479fa3934d2b303aedc57ce4791" Nov 04 01:30:19.481872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.481839 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:19.485482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.485454 4486 generic.go:356] "Generic (PLEG): container finished" podID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerID="709b2cb6d9ee6819eaa148722d40f503e646950685af0c221680a81f2ab12b43" exitCode=60 Nov 04 01:30:19.485609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.485550 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerDied","Data":"709b2cb6d9ee6819eaa148722d40f503e646950685af0c221680a81f2ab12b43"} Nov 04 01:30:19.486018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.485968 4486 scope.go:122] "RemoveContainer" containerID="709b2cb6d9ee6819eaa148722d40f503e646950685af0c221680a81f2ab12b43" Nov 04 01:30:19.486372 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:19.486346 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq_odh-ai-gateway-infra(9695423d-854a-4fac-a1f8-7185c8eb4a4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" Nov 04 01:30:19.488478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.488450 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerStarted","Data":"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a"} Nov 04 01:30:19.488586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.488500 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerStarted","Data":"5b2bfcd6a4ad7e1e40dd3320bfc8cadee20dbb36bc4ce8ec1b02e3f3f9c7322e"} Nov 04 01:30:19.491350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.491330 4486 generic.go:356] "Generic (PLEG): container finished" podID="445d79ac-a030-483b-a248-926b0107bfb9" containerID="288c85865aae34200b089c1b6e7a8dc8753f35349dbbcacd3d9f1338b0e9567b" exitCode=60 Nov 04 01:30:19.491449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.491367 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerDied","Data":"288c85865aae34200b089c1b6e7a8dc8753f35349dbbcacd3d9f1338b0e9567b"} Nov 04 01:30:19.495526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.494995 4486 scope.go:122] "RemoveContainer" containerID="288c85865aae34200b089c1b6e7a8dc8753f35349dbbcacd3d9f1338b0e9567b" Nov 04 01:30:19.495526 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:19.495369 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x_odh-ai-gateway-infra(445d79ac-a030-483b-a248-926b0107bfb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" podUID="445d79ac-a030-483b-a248-926b0107bfb9" Nov 04 01:30:19.499265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.499240 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:19.523370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.517547 4486 scope.go:122] "RemoveContainer" containerID="95f76294c425d3f7bef7d2631fd2f256efcf3a45a652a9e71376555112123473" Nov 04 01:30:19.607298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.605588 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:19.624043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.623485 4486 scope.go:122] "RemoveContainer" containerID="e7f892cfcbb3fe708ade891601d0dbc033c9d384cc078ffa81a98a7c9cffb6e2" Nov 04 01:30:19.682720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.682696 4486 scope.go:122] "RemoveContainer" containerID="b162eae93f60d40f59495828e0c4f1522b346c0f43249f40a835e1d95fa4ca16" Nov 04 01:30:19.704999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.704961 4486 scope.go:122] "RemoveContainer" containerID="def2e80d2d3ce7b7d16a2538cb21f3402e266ececbb08a69f74e685772d7a207" Nov 04 01:30:19.724556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.724422 4486 scope.go:122] "RemoveContainer" containerID="a9c9566ffe2535242fec88b934319179f3b208736aa4a1cbf54d0e24674143e1" Nov 04 01:30:19.807505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.807431 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:19.823341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.823313 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:19.838044 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.828037 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-678aaf-29895930-dj7gh"] Nov 04 01:30:19.923061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.923029 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:19.977780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.977744 4486 scope.go:122] "RemoveContainer" containerID="d3a9b4962d0283cb69aa2179b22b946e6cb3d62ff9531e0c829748a7487959d5" Nov 04 01:30:19.989911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:19.989890 4486 scope.go:122] "RemoveContainer" containerID="c779f0a4a09e10652bd11a18498f0917e40c54df5da119ad0d4aacb78e7cc910" Nov 04 01:30:20.407317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.407283 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:20.413399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.413369 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:20.474865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.474813 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podStartSLOduration=3.474793809 podStartE2EDuration="3.474793809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:20.474629139 +0000 UTC m=+1989.233291301" watchObservedRunningTime="2026-11-04 01:30:20.474793809 +0000 UTC m=+1989.233455967" Nov 04 01:30:20.558009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.556850 4486 generic.go:356] "Generic (PLEG): container finished" podID="888c5373-4a1d-4f15-8769-128f60a0241d" containerID="e36aa4e0cfe88eaed481514c5db7b34609edeb2ca17f2b066b97c8fd8c0243ab" exitCode=60 Nov 04 01:30:20.558009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.556946 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerDied","Data":"e36aa4e0cfe88eaed481514c5db7b34609edeb2ca17f2b066b97c8fd8c0243ab"} Nov 04 01:30:20.558009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.557022 4486 scope.go:122] "RemoveContainer" containerID="1c5dbb54ecf34cf92625f7b8c9d7760860391101b164e896ac00918cb44c09f1" Nov 04 01:30:20.558009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.557526 4486 scope.go:122] "RemoveContainer" containerID="e36aa4e0cfe88eaed481514c5db7b34609edeb2ca17f2b066b97c8fd8c0243ab" Nov 04 01:30:20.558009 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:20.557919 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6_odh-ai-gateway-infra(888c5373-4a1d-4f15-8769-128f60a0241d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" Nov 04 01:30:20.676628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.676604 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:20.682293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.682230 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:30:20.698913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:20.698876 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:21.018063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.017236 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-v6pp8"] Nov 04 01:30:21.127219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.127130 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:21.178935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.178899 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:21.279275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.277650 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\") pod \"766b3721-ffd9-4f93-8cf2-172ccd39887d\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " Nov 04 01:30:21.285105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.281100 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\" (UniqueName: \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\") pod \"766b3721-ffd9-4f93-8cf2-172ccd39887d\" (UID: \"766b3721-ffd9-4f93-8cf2-172ccd39887d\") " Nov 04 01:30:21.285105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.282472 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca" pod "766b3721-ffd9-4f93-8cf2-172ccd39887d" (UID: "766b3721-ffd9-4f93-8cf2-172ccd39887d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.287396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.287363 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb" pod "766b3721-ffd9-4f93-8cf2-172ccd39887d" (UID: "766b3721-ffd9-4f93-8cf2-172ccd39887d"). InnerVolumeSpecName "kube-api-access-5fmvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.316747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.316630 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:21.323116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.323092 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:21.382254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.382225 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\" (UniqueName: \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\") pod \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " Nov 04 01:30:21.382470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.382454 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\") pod \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\" (UID: \"9695423d-854a-4fac-a1f8-7185c8eb4a4f\") " Nov 04 01:30:21.382678 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.382663 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\") pod \"21ccff16-040a-400e-97b2-be728905afa5\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " Nov 04 01:30:21.382925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.382910 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\" (UniqueName: \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\") pod \"21ccff16-040a-400e-97b2-be728905afa5\" (UID: \"21ccff16-040a-400e-97b2-be728905afa5\") " Nov 04 01:30:21.383436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.383064 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca" pod "9695423d-854a-4fac-a1f8-7185c8eb4a4f" (UID: "9695423d-854a-4fac-a1f8-7185c8eb4a4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.383436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.383103 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca" pod "21ccff16-040a-400e-97b2-be728905afa5" (UID: "21ccff16-040a-400e-97b2-be728905afa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.386235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.385652 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ccff16-040a-400e-97b2-be728905afa5-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.386235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.385685 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9695423d-854a-4fac-a1f8-7185c8eb4a4f-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.386235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.385701 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/766b3721-ffd9-4f93-8cf2-172ccd39887d-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.386235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.385715 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmvb\" (UniqueName: \"kubernetes.io/projected/766b3721-ffd9-4f93-8cf2-172ccd39887d-kube-api-access-5fmvb\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.386235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.386192 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc" pod "21ccff16-040a-400e-97b2-be728905afa5" (UID: "21ccff16-040a-400e-97b2-be728905afa5"). InnerVolumeSpecName "kube-api-access-g98bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.391101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.387013 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343e4987-1f43-45b6-b654-497177789217" path="/var/lib/kubelet/pods/343e4987-1f43-45b6-b654-497177789217/volumes" Nov 04 01:30:21.391101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.387326 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:21.391101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.387523 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" path="/var/lib/kubelet/pods/efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff/volumes" Nov 04 01:30:21.391101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.390161 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:21.399068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.397950 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n" pod "9695423d-854a-4fac-a1f8-7185c8eb4a4f" (UID: "9695423d-854a-4fac-a1f8-7185c8eb4a4f"). InnerVolumeSpecName "kube-api-access-5g82n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.486831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.486795 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\" (UniqueName: \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\") pod \"445d79ac-a030-483b-a248-926b0107bfb9\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " Nov 04 01:30:21.487301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487030 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\") pod \"aac273c8-2b61-4570-9a34-eb572bfcedcc\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " Nov 04 01:30:21.487301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487062 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\") pod \"445d79ac-a030-483b-a248-926b0107bfb9\" (UID: \"445d79ac-a030-483b-a248-926b0107bfb9\") " Nov 04 01:30:21.487301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487190 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\" (UniqueName: \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\") pod \"aac273c8-2b61-4570-9a34-eb572bfcedcc\" (UID: \"aac273c8-2b61-4570-9a34-eb572bfcedcc\") " Nov 04 01:30:21.487791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487738 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98bc\" (UniqueName: \"kubernetes.io/projected/21ccff16-040a-400e-97b2-be728905afa5-kube-api-access-g98bc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.487791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487761 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g82n\" (UniqueName: \"kubernetes.io/projected/9695423d-854a-4fac-a1f8-7185c8eb4a4f-kube-api-access-5g82n\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.487970 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.487932 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca" pod "aac273c8-2b61-4570-9a34-eb572bfcedcc" (UID: "aac273c8-2b61-4570-9a34-eb572bfcedcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.488157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.488135 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca" pod "445d79ac-a030-483b-a248-926b0107bfb9" (UID: "445d79ac-a030-483b-a248-926b0107bfb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.489902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.489864 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6" pod "445d79ac-a030-483b-a248-926b0107bfb9" (UID: "445d79ac-a030-483b-a248-926b0107bfb9"). InnerVolumeSpecName "kube-api-access-hbpr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.490128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.489968 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2" pod "aac273c8-2b61-4570-9a34-eb572bfcedcc" (UID: "aac273c8-2b61-4570-9a34-eb572bfcedcc"). InnerVolumeSpecName "kube-api-access-2gbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:21.589083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.589051 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbt2\" (UniqueName: \"kubernetes.io/projected/aac273c8-2b61-4570-9a34-eb572bfcedcc-kube-api-access-2gbt2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.589083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.589079 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbpr6\" (UniqueName: \"kubernetes.io/projected/445d79ac-a030-483b-a248-926b0107bfb9-kube-api-access-hbpr6\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.589083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.589090 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aac273c8-2b61-4570-9a34-eb572bfcedcc-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.589307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.589098 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d79ac-a030-483b-a248-926b0107bfb9-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.685056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.685015 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" event={"ID":"445d79ac-a030-483b-a248-926b0107bfb9","Type":"ContainerDied","Data":"8db8f500f542be1e50b86b883ce598e5618c34f3f3de862041a1f65ca15195c3"} Nov 04 01:30:21.685229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.685086 4486 scope.go:122] "RemoveContainer" containerID="288c85865aae34200b089c1b6e7a8dc8753f35349dbbcacd3d9f1338b0e9567b" Nov 04 01:30:21.685229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.685031 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x" Nov 04 01:30:21.686818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.686799 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" Nov 04 01:30:21.687349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.687325 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts" event={"ID":"766b3721-ffd9-4f93-8cf2-172ccd39887d","Type":"ContainerDied","Data":"1bc2533e528414a84fed264ffe39409ee1c69a31cf27c1c629ead901a67df034"} Nov 04 01:30:21.692154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.692132 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" Nov 04 01:30:21.692243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.692171 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs" event={"ID":"aac273c8-2b61-4570-9a34-eb572bfcedcc","Type":"ContainerDied","Data":"ad4661eb9614f6ff69c0088e1200079813ff5974d51b9af6499c019bb607ea5e"} Nov 04 01:30:21.693930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.693648 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" Nov 04 01:30:21.693930 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.693647 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8" event={"ID":"21ccff16-040a-400e-97b2-be728905afa5","Type":"ContainerDied","Data":"890e7a61986b46e8ecf65211a9d432294c35b2cdf3a081c15608e1016e37a6e4"} Nov 04 01:30:21.696568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.696544 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" event={"ID":"9695423d-854a-4fac-a1f8-7185c8eb4a4f","Type":"ContainerDied","Data":"cf495ed5f1b5ec9e174eb80d87a566fb3ef1805b145051704614644feb046329"} Nov 04 01:30:21.696954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.696936 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq" Nov 04 01:30:21.699860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.699841 4486 scope.go:122] "RemoveContainer" containerID="667b882066b1ba9f9dbaef76e6ce136a8aba15e56ecef8f3cd36b199c02fe888" Nov 04 01:30:21.713024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.712938 4486 scope.go:122] "RemoveContainer" containerID="ec43a70944182292350223314321ba61865c7268f0d27cb84b30b1c2b88d5881" Nov 04 01:30:21.722877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.722619 4486 scope.go:122] "RemoveContainer" containerID="3e9f13b73a50c7856cf9df39d98fb8b5a77a1c5f151d835226778639f0353f40" Nov 04 01:30:21.759034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.758876 4486 scope.go:122] "RemoveContainer" containerID="709b2cb6d9ee6819eaa148722d40f503e646950685af0c221680a81f2ab12b43" Nov 04 01:30:21.880818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.880791 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:21.892566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.892539 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\") pod \"888c5373-4a1d-4f15-8769-128f60a0241d\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " Nov 04 01:30:21.892675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.892588 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\" (UniqueName: \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\") pod \"888c5373-4a1d-4f15-8769-128f60a0241d\" (UID: \"888c5373-4a1d-4f15-8769-128f60a0241d\") " Nov 04 01:30:21.892976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.892952 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca" pod "888c5373-4a1d-4f15-8769-128f60a0241d" (UID: "888c5373-4a1d-4f15-8769-128f60a0241d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:21.893301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.893283 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/888c5373-4a1d-4f15-8769-128f60a0241d-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:21.894809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.894788 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p" pod "888c5373-4a1d-4f15-8769-128f60a0241d" (UID: "888c5373-4a1d-4f15-8769-128f60a0241d"). InnerVolumeSpecName "kube-api-access-wtw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:22.002299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:21.996683 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtw5p\" (UniqueName: \"kubernetes.io/projected/888c5373-4a1d-4f15-8769-128f60a0241d-kube-api-access-wtw5p\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:22.100015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.099955 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:22.107948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.107918 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-98f846-29895930-vfmrq"] Nov 04 01:30:22.271302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.271204 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:22.277521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.277483 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2fd27d-29895930-p872x"] Nov 04 01:30:22.382374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.382343 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:22.386956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.386929 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-5778ad-29895930-bcrts"] Nov 04 01:30:22.679045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.679011 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:22.687867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.687835 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5186fe-29895930-kmrjs"] Nov 04 01:30:22.708238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.708201 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" event={"ID":"888c5373-4a1d-4f15-8769-128f60a0241d","Type":"ContainerDied","Data":"ff2ffaf059b52c3faf33e6bce50f3694cc98d1310697c0fb58d8822df3c3a316"} Nov 04 01:30:22.708238 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.708229 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6" Nov 04 01:30:22.708467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.708296 4486 scope.go:122] "RemoveContainer" containerID="e36aa4e0cfe88eaed481514c5db7b34609edeb2ca17f2b066b97c8fd8c0243ab" Nov 04 01:30:22.875723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.875332 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:22.887910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.887879 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e993d7-29895930-gxpn8"] Nov 04 01:30:22.989477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.989392 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:22.993551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:22.993523 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-42958c-29895930-xrsd6"] Nov 04 01:30:23.354673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.354635 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ccff16-040a-400e-97b2-be728905afa5" path="/var/lib/kubelet/pods/21ccff16-040a-400e-97b2-be728905afa5/volumes" Nov 04 01:30:23.355423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.355401 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445d79ac-a030-483b-a248-926b0107bfb9" path="/var/lib/kubelet/pods/445d79ac-a030-483b-a248-926b0107bfb9/volumes" Nov 04 01:30:23.356074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.356048 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" path="/var/lib/kubelet/pods/766b3721-ffd9-4f93-8cf2-172ccd39887d/volumes" Nov 04 01:30:23.357214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.357186 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" path="/var/lib/kubelet/pods/888c5373-4a1d-4f15-8769-128f60a0241d/volumes" Nov 04 01:30:23.357833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.357809 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" path="/var/lib/kubelet/pods/9695423d-854a-4fac-a1f8-7185c8eb4a4f/volumes" Nov 04 01:30:23.358524 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:23.358499 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" path="/var/lib/kubelet/pods/aac273c8-2b61-4570-9a34-eb572bfcedcc/volumes" Nov 04 01:30:26.765873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.765841 4486 generic.go:356] "Generic (PLEG): container finished" podID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerID="e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb" exitCode=0 Nov 04 01:30:26.766447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.765917 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerDied","Data":"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb"} Nov 04 01:30:26.766447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.766428 4486 scope.go:122] "RemoveContainer" containerID="e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb" Nov 04 01:30:26.768183 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.768150 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:26.768297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.768220 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:26.768297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:26.768251 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:30:27.780253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:27.780217 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerStarted","Data":"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f"} Nov 04 01:30:28.498687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:28.498653 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:30:28.500029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:28.499978 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:28.500103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:28.500057 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 01:30:29.499266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:29.499235 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:29.499666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:29.499286 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 01:30:30.499148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:30.499116 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:30.499325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:30.499170 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 01:30:31.499192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:31.499159 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:31.499385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:31.499220 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 01:30:32.095876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:32.095842 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:30:32.499208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:32.499128 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:32.499208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:32.499175 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 04 01:30:33.503393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:33.503358 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:30:33.504556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:33.504533 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:30:36.260682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.260626 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6"] Nov 04 01:30:36.261589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261567 4486 cpu_manager.go:414] "containerMap: removing container" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261590 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261613 4486 cpu_manager.go:414] "containerMap: removing container" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261621 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261631 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261638 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261653 4486 cpu_manager.go:414] "containerMap: removing container" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261660 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261669 4486 cpu_manager.go:414] "containerMap: removing container" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.261674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261677 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261688 4486 cpu_manager.go:414] "containerMap: removing container" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261695 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261712 4486 cpu_manager.go:414] "containerMap: removing container" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261719 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261736 4486 cpu_manager.go:414] "containerMap: removing container" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261743 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261757 4486 cpu_manager.go:414] "containerMap: removing container" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261764 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261776 4486 cpu_manager.go:414] "containerMap: removing container" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261783 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261799 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261805 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261817 4486 cpu_manager.go:414] "containerMap: removing container" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261824 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261835 4486 cpu_manager.go:414] "containerMap: removing container" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261842 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261858 4486 cpu_manager.go:414] "containerMap: removing container" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261865 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261878 4486 cpu_manager.go:414] "containerMap: removing container" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261885 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261900 4486 cpu_manager.go:414] "containerMap: removing container" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261907 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261918 4486 cpu_manager.go:414] "containerMap: removing container" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261924 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261940 4486 cpu_manager.go:414] "containerMap: removing container" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261947 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261961 4486 cpu_manager.go:414] "containerMap: removing container" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.261968 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262129 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262140 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262151 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262163 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262170 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262182 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262192 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262199 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262209 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262218 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262225 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.262210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262235 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="766b3721-ffd9-4f93-8cf2-172ccd39887d" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262244 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262256 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac273c8-2b61-4570-9a34-eb572bfcedcc" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262262 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262271 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262278 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262287 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:36.264331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.262296 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:36.266327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.266307 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.302637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.302609 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.302795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.302766 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.302875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.302859 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m6n2\" (UniqueName: \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.323036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.323010 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6"] Nov 04 01:30:36.364631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.364602 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:30:36.364836 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.364817 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:30:36.403997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.403956 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.404176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.404145 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.404280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.404257 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m6n2\" (UniqueName: \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.405066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.405043 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.406683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.406658 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.463039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.462976 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m6n2\" (UniqueName: \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.580830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.580800 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:36.967926 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:30:36.967896 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bb0250_b4af_476f_90f3_29d2c175bc35.slice/crio-6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef WatchSource:0}: Error finding container 6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef: Status 404 returned error can't find the container with id 6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef Nov 04 01:30:36.969188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:36.969153 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6"] Nov 04 01:30:37.901148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:37.901074 4486 generic.go:356] "Generic (PLEG): container finished" podID="16bb0250-b4af-476f-90f3-29d2c175bc35" containerID="80d4977950af3a98890447d32c6fb6429be350acc03d5708848ca415609489fb" exitCode=0 Nov 04 01:30:37.901593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:37.901159 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" event={"ID":"16bb0250-b4af-476f-90f3-29d2c175bc35","Type":"ContainerDied","Data":"80d4977950af3a98890447d32c6fb6429be350acc03d5708848ca415609489fb"} Nov 04 01:30:37.901593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:37.901226 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" event={"ID":"16bb0250-b4af-476f-90f3-29d2c175bc35","Type":"ContainerStarted","Data":"6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef"} Nov 04 01:30:39.082873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.082841 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:39.143897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.143864 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\" (UniqueName: \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\") pod \"16bb0250-b4af-476f-90f3-29d2c175bc35\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " Nov 04 01:30:39.144112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.143970 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\") pod \"16bb0250-b4af-476f-90f3-29d2c175bc35\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " Nov 04 01:30:39.144112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.144077 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\") pod \"16bb0250-b4af-476f-90f3-29d2c175bc35\" (UID: \"16bb0250-b4af-476f-90f3-29d2c175bc35\") " Nov 04 01:30:39.144329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.144305 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca" pod "16bb0250-b4af-476f-90f3-29d2c175bc35" (UID: "16bb0250-b4af-476f-90f3-29d2c175bc35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:39.144595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.144577 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:39.146303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.146272 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2" pod "16bb0250-b4af-476f-90f3-29d2c175bc35" (UID: "16bb0250-b4af-476f-90f3-29d2c175bc35"). InnerVolumeSpecName "kube-api-access-8m6n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:39.146403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.146369 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca" pod "16bb0250-b4af-476f-90f3-29d2c175bc35" (UID: "16bb0250-b4af-476f-90f3-29d2c175bc35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:30:39.246512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.246419 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16bb0250-b4af-476f-90f3-29d2c175bc35-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:39.246512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.246457 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m6n2\" (UniqueName: \"kubernetes.io/projected/16bb0250-b4af-476f-90f3-29d2c175bc35-kube-api-access-8m6n2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:39.920153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.920120 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" Nov 04 01:30:39.920329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.920154 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6" event={"ID":"16bb0250-b4af-476f-90f3-29d2c175bc35","Type":"ContainerDied","Data":"6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef"} Nov 04 01:30:39.920329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:39.920218 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c9160fdd076e94188f66ca5557ed4dca36c971cacb8de8a24873ea7c7f873ef" Nov 04 01:30:43.833630 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:30:43.833587 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:30:49.948205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:49.948170 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52"] Nov 04 01:30:49.994904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:49.994872 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9a0432-9tp52"] Nov 04 01:30:51.359689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:51.359646 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21deeee9-3177-4639-866e-f49e78b5593b" path="/var/lib/kubelet/pods/21deeee9-3177-4639-866e-f49e78b5593b/volumes" Nov 04 01:30:57.117508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:57.117421 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf"] Nov 04 01:30:57.164607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:57.164573 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e65b23-5k6gf"] Nov 04 01:30:57.352047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:57.352003 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ae0587-f0b1-47dc-a8de-7cd89c40729d" path="/var/lib/kubelet/pods/33ae0587-f0b1-47dc-a8de-7cd89c40729d/volumes" Nov 04 01:30:58.621938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:58.621905 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:30:58.622377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:58.622124 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" containerID="cri-o://49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a" gracePeriod=30 Nov 04 01:30:59.500724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.500685 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq"] Nov 04 01:30:59.501302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501288 4486 cpu_manager.go:414] "containerMap: removing container" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501303 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343e4987-1f43-45b6-b654-497177789217" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501319 4486 cpu_manager.go:414] "containerMap: removing container" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501325 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501336 4486 cpu_manager.go:414] "containerMap: removing container" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501341 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:59.501353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501352 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501357 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501365 4486 cpu_manager.go:414] "containerMap: removing container" podUID="16bb0250-b4af-476f-90f3-29d2c175bc35" containerName="revoke-keys" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501369 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bb0250-b4af-476f-90f3-29d2c175bc35" containerName="revoke-keys" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501395 4486 cpu_manager.go:414] "containerMap: removing container" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501399 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501481 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9695423d-854a-4fac-a1f8-7185c8eb4a4f" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501488 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d79ac-a030-483b-a248-926b0107bfb9" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501495 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bb0250-b4af-476f-90f3-29d2c175bc35" containerName="revoke-keys" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501499 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="efffc5a4-9ba0-4e53-a2c2-b6a6d1ee57ff" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501506 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="888c5373-4a1d-4f15-8769-128f60a0241d" containerName="cleanup" Nov 04 01:30:59.501540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.501515 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ccff16-040a-400e-97b2-be728905afa5" containerName="cleanup" Nov 04 01:30:59.506428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.506407 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.559655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.559628 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:30:59.559805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.559788 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:30:59.611499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.611446 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq"] Nov 04 01:30:59.630228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.630199 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.630573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.630232 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.630573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.630324 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k998r\" (UniqueName: \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.731604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.731573 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.731604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.731607 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.731857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.731674 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k998r\" (UniqueName: \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.732479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.732450 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.733907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.733884 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.787015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.786943 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k998r\" (UniqueName: \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:30:59.820456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:30:59.820433 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:31:00.199108 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:31:00.199070 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1039cde6_b761_4057_bc2c_9b95da91c23e.slice/crio-686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc WatchSource:0}: Error finding container 686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc: Status 404 returned error can't find the container with id 686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc Nov 04 01:31:00.200689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:00.200560 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq"] Nov 04 01:31:01.100801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:01.100764 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" event={"ID":"1039cde6-b761-4057-bc2c-9b95da91c23e","Type":"ContainerStarted","Data":"94443bccec55f4fa89f68fb9f7ef0ba3a3221b019ff27e719c7530e0f6ba9a26"} Nov 04 01:31:01.101219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:01.100822 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" event={"ID":"1039cde6-b761-4057-bc2c-9b95da91c23e","Type":"ContainerStarted","Data":"686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc"} Nov 04 01:31:01.219003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:01.218925 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" podStartSLOduration=2.218906475 podStartE2EDuration="2.218906475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:01.218066438 +0000 UTC m=+2029.976728597" watchObservedRunningTime="2026-11-04 01:31:01.218906475 +0000 UTC m=+2029.977568841" Nov 04 01:31:01.862599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:01.862559 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:31:01.862780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:01.862624 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:31:02.122635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:02.122533 4486 generic.go:356] "Generic (PLEG): container finished" podID="1039cde6-b761-4057-bc2c-9b95da91c23e" containerID="94443bccec55f4fa89f68fb9f7ef0ba3a3221b019ff27e719c7530e0f6ba9a26" exitCode=0 Nov 04 01:31:02.122635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:02.122577 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" event={"ID":"1039cde6-b761-4057-bc2c-9b95da91c23e","Type":"ContainerDied","Data":"94443bccec55f4fa89f68fb9f7ef0ba3a3221b019ff27e719c7530e0f6ba9a26"} Nov 04 01:31:03.292214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.292189 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:31:03.396588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.396513 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\" (UniqueName: \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\") pod \"1039cde6-b761-4057-bc2c-9b95da91c23e\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " Nov 04 01:31:03.396588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.396546 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\") pod \"1039cde6-b761-4057-bc2c-9b95da91c23e\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " Nov 04 01:31:03.396811 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.396586 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\") pod \"1039cde6-b761-4057-bc2c-9b95da91c23e\" (UID: \"1039cde6-b761-4057-bc2c-9b95da91c23e\") " Nov 04 01:31:03.396971 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.396946 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca" pod "1039cde6-b761-4057-bc2c-9b95da91c23e" (UID: "1039cde6-b761-4057-bc2c-9b95da91c23e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:03.397081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.397061 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:03.398582 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.398561 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca" pod "1039cde6-b761-4057-bc2c-9b95da91c23e" (UID: "1039cde6-b761-4057-bc2c-9b95da91c23e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:03.398720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.398704 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r" pod "1039cde6-b761-4057-bc2c-9b95da91c23e" (UID: "1039cde6-b761-4057-bc2c-9b95da91c23e"). InnerVolumeSpecName "kube-api-access-k998r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:03.498086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.498054 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k998r\" (UniqueName: \"kubernetes.io/projected/1039cde6-b761-4057-bc2c-9b95da91c23e-kube-api-access-k998r\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:03.498086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.498085 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1039cde6-b761-4057-bc2c-9b95da91c23e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:03.923099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.923071 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:31:03.934680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.934630 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6"] Nov 04 01:31:03.943589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:03.943551 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-5186fe-96mh6"] Nov 04 01:31:04.004838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.004802 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.004849 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.004943 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005021 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005083 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005146 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005212 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005278 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005283 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:04.005376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005316 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\") pod \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\" (UID: \"6fcefdeb-c027-4a22-a40f-20f2a5c765c8\") " Nov 04 01:31:04.005529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005391 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:04.005529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005470 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:04.005646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005548 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:04.005727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005706 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:04.005943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005923 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.006006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005951 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.006006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005965 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.006006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.005979 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.006116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.006007 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.007159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.007134 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "kube-api-access-pfmcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:04.007295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.007277 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:04.007500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.007479 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:04.007538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.007479 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy" pod "6fcefdeb-c027-4a22-a40f-20f2a5c765c8" (UID: "6fcefdeb-c027-4a22-a40f-20f2a5c765c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:04.107724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.107685 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfmcw\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-kube-api-access-pfmcw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.107724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.107725 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.107965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.107742 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.107965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.107755 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcefdeb-c027-4a22-a40f-20f2a5c765c8-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:04.111047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.111017 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:31:04.111515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.111487 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" containerID="cri-o://a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e" gracePeriod=30 Nov 04 01:31:04.115513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.115488 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:31:04.115684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.115664 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" containerID="cri-o://ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11" gracePeriod=30 Nov 04 01:31:04.121106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.121085 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:31:04.121249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.121233 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" containerID="cri-o://17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9" gracePeriod=30 Nov 04 01:31:04.140653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.140628 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" Nov 04 01:31:04.140759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.140687 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq" event={"ID":"1039cde6-b761-4057-bc2c-9b95da91c23e","Type":"ContainerDied","Data":"686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc"} Nov 04 01:31:04.140759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.140738 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686e8ed6931a21d9ee8c9937658aab23db6feca02cc093c82ddda6b5ac7f26cc" Nov 04 01:31:04.142720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.142696 4486 generic.go:356] "Generic (PLEG): container finished" podID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerID="49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a" exitCode=0 Nov 04 01:31:04.142840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.142757 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerDied","Data":"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a"} Nov 04 01:31:04.142840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.142762 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" Nov 04 01:31:04.142840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.142789 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2" event={"ID":"6fcefdeb-c027-4a22-a40f-20f2a5c765c8","Type":"ContainerDied","Data":"43d5fa80134559c00785713e8291266a3d054470eeff355e0a60f91078c1ac50"} Nov 04 01:31:04.142840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.142810 4486 scope.go:122] "RemoveContainer" containerID="49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a" Nov 04 01:31:04.163188 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.163145 4486 scope.go:122] "RemoveContainer" containerID="fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285" Nov 04 01:31:04.173303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.173287 4486 scope.go:122] "RemoveContainer" containerID="49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a" Nov 04 01:31:04.173570 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:04.173552 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a\": container with ID starting with 49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a not found: ID does not exist" containerID="49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a" Nov 04 01:31:04.173632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.173585 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a"} err="failed to get container status \"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a\": rpc error: code = NotFound desc = could not find container \"49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a\": container with ID starting with 49f88fe0cfcf8107dc63f0ddf6f9e68f4fdd2ba4b55ae61015181eb14b68a74a not found: ID does not exist" Nov 04 01:31:04.173632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.173602 4486 scope.go:122] "RemoveContainer" containerID="fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285" Nov 04 01:31:04.173820 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:04.173805 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285\": container with ID starting with fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285 not found: ID does not exist" containerID="fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285" Nov 04 01:31:04.173857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.173824 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285"} err="failed to get container status \"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285\": rpc error: code = NotFound desc = could not find container \"fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285\": container with ID starting with fb6c0781555564fbb277f2194785ffc57c197195fdce1ba1f18e872bc94c8285 not found: ID does not exist" Nov 04 01:31:04.185396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.185367 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:31:04.185489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.185409 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:31:04.644851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.644813 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:31:04.655283 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.655250 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9a0432-openshift-default-cff994797-8cnl2"] Nov 04 01:31:04.810752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.810715 4486 patch_prober.go:29] interesting pod/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:31:04.810924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:04.810764 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:31:05.354241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:05.354205 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bb0250-b4af-476f-90f3-29d2c175bc35" path="/var/lib/kubelet/pods/16bb0250-b4af-476f-90f3-29d2c175bc35/volumes" Nov 04 01:31:05.354790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:05.354767 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" path="/var/lib/kubelet/pods/6fcefdeb-c027-4a22-a40f-20f2a5c765c8/volumes" Nov 04 01:31:07.027454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.027415 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:31:07.028071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028049 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1039cde6-b761-4057-bc2c-9b95da91c23e" containerName="revoke-keys" Nov 04 01:31:07.028071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028070 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1039cde6-b761-4057-bc2c-9b95da91c23e" containerName="revoke-keys" Nov 04 01:31:07.028179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028104 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.028179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028109 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.028179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028119 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.028179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028124 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.028296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028211 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1039cde6-b761-4057-bc2c-9b95da91c23e" containerName="revoke-keys" Nov 04 01:31:07.028296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028225 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.028296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.028233 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcefdeb-c027-4a22-a40f-20f2a5c765c8" containerName="istio-proxy" Nov 04 01:31:07.033268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.033239 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.091292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.091262 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-843be0-openshift-default-dockercfg-dhchd\"" Nov 04 01:31:07.145378 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.145341 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:31:07.162336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162300 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162367 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162396 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbchs\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162515 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162623 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162642 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162661 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162754 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.162909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.162888 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.263946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.263896 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.263974 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264039 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264068 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbchs\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264145 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264257 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264286 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264313 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264374 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264389 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264527 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264627 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264697 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.264884 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.264865 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.266427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.266401 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.266580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.266561 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.321297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.321272 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.323099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.323081 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbchs\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\") pod \"e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.351262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.351237 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:07.817012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.816948 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:31:07.822677 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:31:07.822645 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a87968_6e6d_4188_a8d5_1d03cc748fba.slice/crio-9e8ba61ace4d4daf94ce991742280a36eb73aec04ce4e8f3483c6f48bbe18b78 WatchSource:0}: Error finding container 9e8ba61ace4d4daf94ce991742280a36eb73aec04ce4e8f3483c6f48bbe18b78: Status 404 returned error can't find the container with id 9e8ba61ace4d4daf94ce991742280a36eb73aec04ce4e8f3483c6f48bbe18b78 Nov 04 01:31:07.825224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.825181 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:07.825325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.825257 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:07.825325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:07.825286 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:08.184869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:08.184782 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerStarted","Data":"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f"} Nov 04 01:31:08.184869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:08.184839 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerStarted","Data":"9e8ba61ace4d4daf94ce991742280a36eb73aec04ce4e8f3483c6f48bbe18b78"} Nov 04 01:31:08.323522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:08.323472 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podStartSLOduration=2.3234583300000002 podStartE2EDuration="2.32345833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:08.319681376 +0000 UTC m=+2037.078343535" watchObservedRunningTime="2026-11-04 01:31:08.32345833 +0000 UTC m=+2037.082120489" Nov 04 01:31:08.351387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:08.351357 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:08.357855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:08.357826 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:09.193349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.193321 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:09.194212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.194189 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:31:09.434529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.434125 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:31:09.507935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.507901 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508033 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508095 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508147 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508368 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508436 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508491 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508527 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.508636 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.508597 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\") pod \"440f9d59-95a2-492f-af05-cc83fd171f83\" (UID: \"440f9d59-95a2-492f-af05-cc83fd171f83\") " Nov 04 01:31:09.509468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.509441 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.509808 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.509774 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.510156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.510126 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:09.510394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.510370 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.514292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.514103 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.514386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.514329 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.515255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.515227 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.515348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.515335 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "kube-api-access-ph8sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.515405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.515394 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo" pod "440f9d59-95a2-492f-af05-cc83fd171f83" (UID: "440f9d59-95a2-492f-af05-cc83fd171f83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:09.556015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.552698 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:31:09.556015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.555635 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:31:09.609724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609695 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.609917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609744 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.609917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609789 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.609917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609864 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.609917 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609901 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609938 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.609960 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610103 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.610168 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610150 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610221 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610276 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610328 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610367 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610419 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610450 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610479 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\") pod \"9ee6b30f-f5ca-4118-8132-d4328116365b\" (UID: \"9ee6b30f-f5ca-4118-8132-d4328116365b\") " Nov 04 01:31:09.610612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610553 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610594 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.610852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.610654 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\") pod \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\" (UID: \"73edaad7-e2b6-4b23-89b6-c8d78566dbdf\") " Nov 04 01:31:09.611309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611279 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.611468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611446 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611471 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611490 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8sw\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-kube-api-access-ph8sw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611506 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611521 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611535 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611549 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440f9d59-95a2-492f-af05-cc83fd171f83-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611564 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440f9d59-95a2-492f-af05-cc83fd171f83-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611578 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440f9d59-95a2-492f-af05-cc83fd171f83-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611592 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611641 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440f9d59-95a2-492f-af05-cc83fd171f83-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.611880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611835 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.611999 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:09.612256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612102 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612303 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612319 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612502 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612716 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.612851 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.612784 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:09.617935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.617814 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.618422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.618398 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:09.621496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.621468 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "kube-api-access-7jfbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.622303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.622282 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.623315 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.623287 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.623429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.623355 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv" pod "73edaad7-e2b6-4b23-89b6-c8d78566dbdf" (UID: "73edaad7-e2b6-4b23-89b6-c8d78566dbdf"). InnerVolumeSpecName "kube-api-access-zrkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:09.623429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.623401 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:09.624555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.624534 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo" pod "9ee6b30f-f5ca-4118-8132-d4328116365b" (UID: "9ee6b30f-f5ca-4118-8132-d4328116365b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:09.712616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712546 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712579 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712595 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712609 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkwv\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-kube-api-access-zrkwv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712624 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712638 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712651 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfbb\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-kube-api-access-7jfbb\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712663 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712672 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712680 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712688 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712698 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee6b30f-f5ca-4118-8132-d4328116365b-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712706 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712714 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73edaad7-e2b6-4b23-89b6-c8d78566dbdf-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712722 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:09.712877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:09.712730 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee6b30f-f5ca-4118-8132-d4328116365b-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:10.202729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.202687 4486 generic.go:356] "Generic (PLEG): container finished" podID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerID="ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11" exitCode=0 Nov 04 01:31:10.203247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.202742 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" Nov 04 01:31:10.203247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.202768 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" event={"ID":"73edaad7-e2b6-4b23-89b6-c8d78566dbdf","Type":"ContainerDied","Data":"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11"} Nov 04 01:31:10.203247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.202834 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk" event={"ID":"73edaad7-e2b6-4b23-89b6-c8d78566dbdf","Type":"ContainerDied","Data":"7627d0b13ac1a5e77dffa7938510b455d8728e058f2325fe2a7bf6b1cd3a80d8"} Nov 04 01:31:10.203247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.202866 4486 scope.go:122] "RemoveContainer" containerID="ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11" Nov 04 01:31:10.204864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.204836 4486 generic.go:356] "Generic (PLEG): container finished" podID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerID="a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e" exitCode=0 Nov 04 01:31:10.204994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.204894 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" Nov 04 01:31:10.204994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.204912 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerDied","Data":"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e"} Nov 04 01:31:10.204994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.204965 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l" event={"ID":"9ee6b30f-f5ca-4118-8132-d4328116365b","Type":"ContainerDied","Data":"a154e8b5a9b2404ac4863a09eecc0903ea56e5dd04302f957064619fb8cb7851"} Nov 04 01:31:10.206665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.206639 4486 generic.go:356] "Generic (PLEG): container finished" podID="440f9d59-95a2-492f-af05-cc83fd171f83" containerID="17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9" exitCode=0 Nov 04 01:31:10.206794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.206681 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" Nov 04 01:31:10.206794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.206730 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" event={"ID":"440f9d59-95a2-492f-af05-cc83fd171f83","Type":"ContainerDied","Data":"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9"} Nov 04 01:31:10.206794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.206778 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2" event={"ID":"440f9d59-95a2-492f-af05-cc83fd171f83","Type":"ContainerDied","Data":"f47f87511a6c3c0456da46dc6a739c7ec0fb512ee124d68f1802cc1233742361"} Nov 04 01:31:10.235679 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.235596 4486 scope.go:122] "RemoveContainer" containerID="ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11" Nov 04 01:31:10.236079 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:10.236045 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11\": container with ID starting with ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11 not found: ID does not exist" containerID="ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11" Nov 04 01:31:10.236152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.236091 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11"} err="failed to get container status \"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11\": rpc error: code = NotFound desc = could not find container \"ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11\": container with ID starting with ad68c2ba1e193db6fdccf92ddd3a14585d19afeeb71506ff402d5739fa7cac11 not found: ID does not exist" Nov 04 01:31:10.236152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.236117 4486 scope.go:122] "RemoveContainer" containerID="a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e" Nov 04 01:31:10.249572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.249546 4486 scope.go:122] "RemoveContainer" containerID="e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc" Nov 04 01:31:10.260547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.260512 4486 scope.go:122] "RemoveContainer" containerID="a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e" Nov 04 01:31:10.260811 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:10.260791 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e\": container with ID starting with a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e not found: ID does not exist" containerID="a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e" Nov 04 01:31:10.260898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.260819 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e"} err="failed to get container status \"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e\": rpc error: code = NotFound desc = could not find container \"a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e\": container with ID starting with a5d7345d2f69f3cc98545520aa705fe435ce55f3237854833b73fdabfe3e9c4e not found: ID does not exist" Nov 04 01:31:10.260898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.260835 4486 scope.go:122] "RemoveContainer" containerID="e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc" Nov 04 01:31:10.261121 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:10.261096 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc\": container with ID starting with e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc not found: ID does not exist" containerID="e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc" Nov 04 01:31:10.261198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.261117 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc"} err="failed to get container status \"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc\": rpc error: code = NotFound desc = could not find container \"e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc\": container with ID starting with e337f82d4e511f41f2c52b0edd31263f819cf0f4376ec9900a097e5231fa9dbc not found: ID does not exist" Nov 04 01:31:10.261198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.261130 4486 scope.go:122] "RemoveContainer" containerID="17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9" Nov 04 01:31:10.270887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.270866 4486 scope.go:122] "RemoveContainer" containerID="17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9" Nov 04 01:31:10.271205 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:10.271184 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9\": container with ID starting with 17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9 not found: ID does not exist" containerID="17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9" Nov 04 01:31:10.271292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.271211 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9"} err="failed to get container status \"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9\": rpc error: code = NotFound desc = could not find container \"17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9\": container with ID starting with 17973f0ecdb7eca1d824c60670a787619a3b0a362452debd93dc4b8d52c7c4b9 not found: ID does not exist" Nov 04 01:31:10.373198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.373162 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:31:10.417902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.417872 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-9pr7l"] Nov 04 01:31:10.537505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.537430 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:31:10.545294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.545263 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-mzgwk"] Nov 04 01:31:10.655766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.655736 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:31:10.666572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:10.666544 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e65b23-openshift-default-6d67df5759-959h2"] Nov 04 01:31:11.352308 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:11.352274 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" path="/var/lib/kubelet/pods/440f9d59-95a2-492f-af05-cc83fd171f83/volumes" Nov 04 01:31:11.352911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:11.352891 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" path="/var/lib/kubelet/pods/73edaad7-e2b6-4b23-89b6-c8d78566dbdf/volumes" Nov 04 01:31:11.353520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:11.353501 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" path="/var/lib/kubelet/pods/9ee6b30f-f5ca-4118-8132-d4328116365b/volumes" Nov 04 01:31:13.926172 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:13.926129 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:31:14.646332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:14.646297 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:31:14.646548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:14.646526 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" containerID="cri-o://c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9" gracePeriod=30 Nov 04 01:31:16.879635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.879596 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq"] Nov 04 01:31:16.880350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880332 4486 cpu_manager.go:414] "containerMap: removing container" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" Nov 04 01:31:16.880401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880352 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" Nov 04 01:31:16.880401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880365 4486 cpu_manager.go:414] "containerMap: removing container" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" Nov 04 01:31:16.880401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880370 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" Nov 04 01:31:16.880401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880390 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.880401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880396 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880408 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880413 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880507 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="73edaad7-e2b6-4b23-89b6-c8d78566dbdf" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880514 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="440f9d59-95a2-492f-af05-cc83fd171f83" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880523 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.880563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.880531 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee6b30f-f5ca-4118-8132-d4328116365b" containerName="istio-proxy" Nov 04 01:31:16.884019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.883999 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:16.890360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.890337 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:31:16.893269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.893250 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:16.945900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.945865 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-843be0\"" Nov 04 01:31:16.986669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.986326 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-843be0-dockercfg-k92s5\"" Nov 04 01:31:16.986669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.986580 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:31:16.986669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.986635 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:31:16.997667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:16.997563 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq"] Nov 04 01:31:17.019236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.014288 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:31:17.044213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.044178 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:31:17.054502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.054465 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.054881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.054848 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.058973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.054917 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.059206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.059175 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tm2\" (UniqueName: \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.059310 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.059244 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j9j\" (UniqueName: \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.059376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.059319 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.084175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.084141 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:31:17.161148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161054 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.161148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161125 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.161391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161286 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.161391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161362 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.161506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161428 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tm2\" (UniqueName: \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.161506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161473 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hnc\" (UniqueName: \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.161798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.161681 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j9j\" (UniqueName: \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.162139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.162113 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.162257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.162208 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.163933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.163910 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.219224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.219196 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j9j\" (UniqueName: \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.219691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.219664 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tm2\" (UniqueName: \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\") pod \"payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.263264 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.263231 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.263420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.263391 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hnc\" (UniqueName: \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.263963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.263935 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.323799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.323775 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hnc\" (UniqueName: \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\") pod \"payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.378961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.378933 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:17.497150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.497107 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:17.506724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.506701 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:17.758673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.758640 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:31:17.876053 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:31:17.876025 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d1b749_01f0_44d4_8fdc_d0c50c127b02.slice/crio-1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f WatchSource:0}: Error finding container 1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f: Status 404 returned error can't find the container with id 1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f Nov 04 01:31:17.877588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.877564 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq"] Nov 04 01:31:17.911247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:17.909418 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:31:17.920856 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:31:17.920830 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e227e1_8243_46fb_aa11_b4170b24c321.slice/crio-8b71383479b9c3a76dc627ac4b934a67a88332ffae32d728e1d9022738a6e18e WatchSource:0}: Error finding container 8b71383479b9c3a76dc627ac4b934a67a88332ffae32d728e1d9022738a6e18e: Status 404 returned error can't find the container with id 8b71383479b9c3a76dc627ac4b934a67a88332ffae32d728e1d9022738a6e18e Nov 04 01:31:18.273393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.273356 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" event={"ID":"d4d1b749-01f0-44d4-8fdc-d0c50c127b02","Type":"ContainerStarted","Data":"ba5693b8bb1d583255c7254893aa5714f8b6d0db131aac98598defd74efe3e24"} Nov 04 01:31:18.273534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.273435 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" event={"ID":"d4d1b749-01f0-44d4-8fdc-d0c50c127b02","Type":"ContainerStarted","Data":"1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f"} Nov 04 01:31:18.275117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.275093 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" event={"ID":"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6","Type":"ContainerStarted","Data":"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda"} Nov 04 01:31:18.275234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.275136 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" event={"ID":"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6","Type":"ContainerStarted","Data":"be3e676da219a04d00d7279d53478ceb7a49be443b2f08131bbb8b5ba6d10f1f"} Nov 04 01:31:18.275234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.275173 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:18.277125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.277060 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" event={"ID":"10e227e1-8243-46fb-aa11-b4170b24c321","Type":"ContainerStarted","Data":"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4"} Nov 04 01:31:18.277125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.277117 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" event={"ID":"10e227e1-8243-46fb-aa11-b4170b24c321","Type":"ContainerStarted","Data":"8b71383479b9c3a76dc627ac4b934a67a88332ffae32d728e1d9022738a6e18e"} Nov 04 01:31:18.277278 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.277172 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:18.396049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.395888 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" podStartSLOduration=2.395870546 podStartE2EDuration="2.395870546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:18.394072627 +0000 UTC m=+2047.152734789" watchObservedRunningTime="2026-11-04 01:31:18.395870546 +0000 UTC m=+2047.154532709" Nov 04 01:31:18.512300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.512249 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" podStartSLOduration=2.5122349010000002 podStartE2EDuration="2.512234901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:18.50904777 +0000 UTC m=+2047.267709930" watchObservedRunningTime="2026-11-04 01:31:18.512234901 +0000 UTC m=+2047.270897061" Nov 04 01:31:18.634890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:18.634823 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" podStartSLOduration=2.634800289 podStartE2EDuration="2.634800289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:18.628373291 +0000 UTC m=+2047.387035456" watchObservedRunningTime="2026-11-04 01:31:18.634800289 +0000 UTC m=+2047.393462465" Nov 04 01:31:19.287547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.287456 4486 generic.go:356] "Generic (PLEG): container finished" podID="d4d1b749-01f0-44d4-8fdc-d0c50c127b02" containerID="ba5693b8bb1d583255c7254893aa5714f8b6d0db131aac98598defd74efe3e24" exitCode=0 Nov 04 01:31:19.288049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.287540 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" event={"ID":"d4d1b749-01f0-44d4-8fdc-d0c50c127b02","Type":"ContainerDied","Data":"ba5693b8bb1d583255c7254893aa5714f8b6d0db131aac98598defd74efe3e24"} Nov 04 01:31:19.924154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.924092 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:31:19.999407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999374 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999429 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999461 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999511 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999557 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999601 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999639 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999663 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:19.999830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999733 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:19.999830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999742 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\") pod \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\" (UID: \"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c\") " Nov 04 01:31:20.000126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999786 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:20.000126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:19.999895 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:20.000244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000188 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:20.000244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000208 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:20.000420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000405 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.000458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000427 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.000458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000440 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.000458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000450 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.000563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.000458 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.002373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.002342 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "kube-api-access-dhjn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:20.002373 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.002351 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:20.002549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.002394 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:20.002549 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.002460 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy" pod "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" (UID: "9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:20.102226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.102177 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjn2\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-kube-api-access-dhjn2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.102226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.102228 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.102469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.102245 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.102469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.102261 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.113608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.113566 4486 scope.go:122] "RemoveContainer" containerID="8b0a318cc4ba393840496bda6423b061b1e5cae8adcc82c6c0cf370479c36bc9" Nov 04 01:31:20.127631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.127611 4486 scope.go:122] "RemoveContainer" containerID="30b311e5a4a1b87c4c8554843dbb40b8c62d8078016cdf18ad3cc70e56529d16" Nov 04 01:31:20.296170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.296133 4486 generic.go:356] "Generic (PLEG): container finished" podID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerID="c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9" exitCode=0 Nov 04 01:31:20.296555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.296201 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" Nov 04 01:31:20.296555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.296219 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerDied","Data":"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9"} Nov 04 01:31:20.296555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.296289 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f" event={"ID":"9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c","Type":"ContainerDied","Data":"82d356411a6cab0907dc022a1a9f57bc9a5c0cb52be7375428931072fa424f76"} Nov 04 01:31:20.296555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.296311 4486 scope.go:122] "RemoveContainer" containerID="c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9" Nov 04 01:31:20.310257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.310089 4486 scope.go:122] "RemoveContainer" containerID="873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6" Nov 04 01:31:20.320533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.320509 4486 scope.go:122] "RemoveContainer" containerID="c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9" Nov 04 01:31:20.320800 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:20.320782 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9\": container with ID starting with c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9 not found: ID does not exist" containerID="c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9" Nov 04 01:31:20.320869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.320812 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9"} err="failed to get container status \"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9\": rpc error: code = NotFound desc = could not find container \"c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9\": container with ID starting with c739c575dd21a61a6b5df716b240063154c7e152eae8caf1ea99be52c9f1fbe9 not found: ID does not exist" Nov 04 01:31:20.320869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.320830 4486 scope.go:122] "RemoveContainer" containerID="873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6" Nov 04 01:31:20.321099 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:20.321079 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6\": container with ID starting with 873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6 not found: ID does not exist" containerID="873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6" Nov 04 01:31:20.321156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.321101 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6"} err="failed to get container status \"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6\": rpc error: code = NotFound desc = could not find container \"873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6\": container with ID starting with 873e6c084f93c66544e15760df7252134031074b7779093724a3ac7e64761da6 not found: ID does not exist" Nov 04 01:31:20.428589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.428532 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:20.457906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.457872 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:31:20.505767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.505730 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5186fe-openshift-default-54995fdf7-qnr8f"] Nov 04 01:31:20.508513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.508489 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\" (UniqueName: \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\") pod \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " Nov 04 01:31:20.508669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.508526 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\") pod \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " Nov 04 01:31:20.508669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.508559 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\") pod \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\" (UID: \"d4d1b749-01f0-44d4-8fdc-d0c50c127b02\") " Nov 04 01:31:20.508961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.508934 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca" pod "d4d1b749-01f0-44d4-8fdc-d0c50c127b02" (UID: "d4d1b749-01f0-44d4-8fdc-d0c50c127b02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:20.509334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.509306 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.510698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.510674 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca" pod "d4d1b749-01f0-44d4-8fdc-d0c50c127b02" (UID: "d4d1b749-01f0-44d4-8fdc-d0c50c127b02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:20.510797 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.510756 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j" pod "d4d1b749-01f0-44d4-8fdc-d0c50c127b02" (UID: "d4d1b749-01f0-44d4-8fdc-d0c50c127b02"). InnerVolumeSpecName "kube-api-access-c5j9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:20.610155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.610121 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j9j\" (UniqueName: \"kubernetes.io/projected/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-kube-api-access-c5j9j\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:20.610155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:20.610148 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d1b749-01f0-44d4-8fdc-d0c50c127b02-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:21.307290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:21.307258 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" Nov 04 01:31:21.307946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:21.307290 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq" event={"ID":"d4d1b749-01f0-44d4-8fdc-d0c50c127b02","Type":"ContainerDied","Data":"1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f"} Nov 04 01:31:21.307946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:21.307345 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d18636b61ff5d64492530338006d09a8673db25dcf1694b07ce567a4223951f" Nov 04 01:31:21.350627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:21.350598 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" path="/var/lib/kubelet/pods/9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c/volumes" Nov 04 01:31:29.294351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:29.294317 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:31:29.331709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:29.323248 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:31:42.534140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.534050 4486 generic.go:356] "Generic (PLEG): container finished" podID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerID="020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a" exitCode=0 Nov 04 01:31:42.534140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.534122 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerDied","Data":"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a"} Nov 04 01:31:42.534627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.534511 4486 scope.go:122] "RemoveContainer" containerID="020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a" Nov 04 01:31:42.536043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.536017 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:42.536144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.536089 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:42.536144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:42.536124 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:31:43.559685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:43.559638 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerStarted","Data":"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3"} Nov 04 01:31:43.862381 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:31:43.862339 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:31:44.407419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:44.407389 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:31:44.407601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:44.407562 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:31:44.407655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:44.407618 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 01:31:45.407367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:45.407325 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:31:45.407814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:45.407400 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 01:31:46.426009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:46.414530 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:31:46.426009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:46.414592 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 01:31:47.407418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:47.407363 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:31:47.409297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:47.408825 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 01:31:48.366768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:48.366732 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq"] Nov 04 01:31:48.407162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:48.407123 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:31:48.407334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:48.407194 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Nov 04 01:31:48.413778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:48.413749 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3dc6c8-9d2hq"] Nov 04 01:31:49.349820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:49.349789 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1039cde6-b761-4057-bc2c-9b95da91c23e" path="/var/lib/kubelet/pods/1039cde6-b761-4057-bc2c-9b95da91c23e/volumes" Nov 04 01:31:49.411467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:49.411442 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:31:49.411927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:49.411854 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:31:49.412642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:49.412620 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:31:55.691522 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:55.691485 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:31:55.691972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:55.691732 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" containerID="cri-o://ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990" gracePeriod=30 Nov 04 01:31:57.166684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.166641 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz"] Nov 04 01:31:57.168669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.168637 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:31:57.168832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.168818 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:31:57.168964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.168953 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d4d1b749-01f0-44d4-8fdc-d0c50c127b02" containerName="revoke-keys" Nov 04 01:31:57.169070 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.169059 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d1b749-01f0-44d4-8fdc-d0c50c127b02" containerName="revoke-keys" Nov 04 01:31:57.169464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.169445 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:31:57.169593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.169581 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d1b749-01f0-44d4-8fdc-d0c50c127b02" containerName="revoke-keys" Nov 04 01:31:57.169685 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.169675 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:31:57.177878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.177851 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.224629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.224599 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz"] Nov 04 01:31:57.266109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.266080 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:31:57.266273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.266105 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:31:57.285632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.285603 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4l4\" (UniqueName: \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.285775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.285754 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.285820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.285797 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.387187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.387156 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4l4\" (UniqueName: \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.387364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.387297 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.387364 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.387345 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.388111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.388090 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.389847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.389817 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.443202 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.443140 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4l4\" (UniqueName: \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\") pod \"maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.490743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.490719 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:31:57.873389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:57.873353 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz"] Nov 04 01:31:57.876197 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:31:57.876169 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34c2c1_34c8_4938_9158_c11eb475e640.slice/crio-b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9 WatchSource:0}: Error finding container b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9: Status 404 returned error can't find the container with id b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9 Nov 04 01:31:58.689647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:58.689610 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" event={"ID":"5a34c2c1-34c8-4938-9158-c11eb475e640","Type":"ContainerStarted","Data":"e0d260dc5f20209bc5880fa54e7fa71ac6b22fa01bf5171a905d73568f0f1c4c"} Nov 04 01:31:58.690148 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:58.689669 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" event={"ID":"5a34c2c1-34c8-4938-9158-c11eb475e640","Type":"ContainerStarted","Data":"b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9"} Nov 04 01:31:58.803023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:58.802961 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" podStartSLOduration=1.8029460510000002 podStartE2EDuration="1.802946051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:58.801112116 +0000 UTC m=+2087.559774288" watchObservedRunningTime="2026-11-04 01:31:58.802946051 +0000 UTC m=+2087.561608223" Nov 04 01:31:59.699325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:59.699290 4486 generic.go:356] "Generic (PLEG): container finished" podID="5a34c2c1-34c8-4938-9158-c11eb475e640" containerID="e0d260dc5f20209bc5880fa54e7fa71ac6b22fa01bf5171a905d73568f0f1c4c" exitCode=0 Nov 04 01:31:59.699772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:31:59.699378 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" event={"ID":"5a34c2c1-34c8-4938-9158-c11eb475e640","Type":"ContainerDied","Data":"e0d260dc5f20209bc5880fa54e7fa71ac6b22fa01bf5171a905d73568f0f1c4c"} Nov 04 01:32:00.868725 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.868697 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:32:00.934560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.934530 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\") pod \"5a34c2c1-34c8-4938-9158-c11eb475e640\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " Nov 04 01:32:00.934762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.934663 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\" (UniqueName: \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\") pod \"5a34c2c1-34c8-4938-9158-c11eb475e640\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " Nov 04 01:32:00.934762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.934681 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\") pod \"5a34c2c1-34c8-4938-9158-c11eb475e640\" (UID: \"5a34c2c1-34c8-4938-9158-c11eb475e640\") " Nov 04 01:32:00.935009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.934956 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca" pod "5a34c2c1-34c8-4938-9158-c11eb475e640" (UID: "5a34c2c1-34c8-4938-9158-c11eb475e640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:00.937207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.937180 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4" pod "5a34c2c1-34c8-4938-9158-c11eb475e640" (UID: "5a34c2c1-34c8-4938-9158-c11eb475e640"). InnerVolumeSpecName "kube-api-access-hh4l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:00.938421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.938396 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca" pod "5a34c2c1-34c8-4938-9158-c11eb475e640" (UID: "5a34c2c1-34c8-4938-9158-c11eb475e640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:00.967764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:00.967745 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036374 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036510 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036562 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036606 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036709 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036757 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036833 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.036888 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.037025 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\") pod \"d5224ab9-4358-481d-b5ba-c7603a5266fd\" (UID: \"d5224ab9-4358-481d-b5ba-c7603a5266fd\") " Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.037801 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.037826 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh4l4\" (UniqueName: \"kubernetes.io/projected/5a34c2c1-34c8-4938-9158-c11eb475e640-kube-api-access-hh4l4\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.037844 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a34c2c1-34c8-4938-9158-c11eb475e640-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.038724 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.039053 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.039170 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.039221 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.039412 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:01.041041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.040853 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "kube-api-access-njlmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:01.042195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.041515 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:01.042593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.042566 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:01.043261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.043235 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token" pod "d5224ab9-4358-481d-b5ba-c7603a5266fd" (UID: "d5224ab9-4358-481d-b5ba-c7603a5266fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:01.139211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139179 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139208 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139219 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139228 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139238 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139247 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlmx\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-kube-api-access-njlmx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139255 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5224ab9-4358-481d-b5ba-c7603a5266fd-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139265 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5224ab9-4358-481d-b5ba-c7603a5266fd-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.139461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.139274 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5224ab9-4358-481d-b5ba-c7603a5266fd-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:01.167203 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.167169 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:32:01.723135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.723107 4486 generic.go:356] "Generic (PLEG): container finished" podID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerID="ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990" exitCode=0 Nov 04 01:32:01.723261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.723178 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" Nov 04 01:32:01.723261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.723196 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerDied","Data":"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990"} Nov 04 01:32:01.723355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.723291 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" event={"ID":"d5224ab9-4358-481d-b5ba-c7603a5266fd","Type":"ContainerDied","Data":"c132b5f1a0cfdd9a136719854360ca6321c6f3c2de7017484922867a62d1bda4"} Nov 04 01:32:01.723355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.723327 4486 scope.go:122] "RemoveContainer" containerID="ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990" Nov 04 01:32:01.725146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.725123 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" Nov 04 01:32:01.725686 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.725117 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz" event={"ID":"5a34c2c1-34c8-4938-9158-c11eb475e640","Type":"ContainerDied","Data":"b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9"} Nov 04 01:32:01.725773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.725707 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24276c1cbccd8b2fe19ec7a53887b4d430518d31fade6eae4fe60b6949169f9" Nov 04 01:32:01.738957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.738873 4486 scope.go:122] "RemoveContainer" containerID="6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564" Nov 04 01:32:01.749341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.749323 4486 scope.go:122] "RemoveContainer" containerID="ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990" Nov 04 01:32:01.749639 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:01.749619 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990\": container with ID starting with ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990 not found: ID does not exist" containerID="ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990" Nov 04 01:32:01.749698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.749647 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990"} err="failed to get container status \"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990\": rpc error: code = NotFound desc = could not find container \"ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990\": container with ID starting with ac3f7026b8d7b3733b3015336321b46ef58b843d52b9fe7a6c46e76f761d2990 not found: ID does not exist" Nov 04 01:32:01.749748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.749700 4486 scope.go:122] "RemoveContainer" containerID="6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564" Nov 04 01:32:01.750047 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:01.749976 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564\": container with ID starting with 6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564 not found: ID does not exist" containerID="6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564" Nov 04 01:32:01.750121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.750051 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564"} err="failed to get container status \"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564\": rpc error: code = NotFound desc = could not find container \"6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564\": container with ID starting with 6801063de5d09663e1dea3693768af5da358e4ed7c9bc24bda07ae11af58b564 not found: ID does not exist" Nov 04 01:32:01.911739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.911701 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:01.912159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.911761 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:01.937063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.937028 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:32:01.941795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:01.941771 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3dc6c8-openshift-default-7d9d8cfb67-4lch7"] Nov 04 01:32:03.352258 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:03.352223 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" path="/var/lib/kubelet/pods/d5224ab9-4358-481d-b5ba-c7603a5266fd/volumes" Nov 04 01:32:07.792887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:07.792846 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:32:07.912443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:07.912409 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:32:08.354957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.354923 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr"] Nov 04 01:32:08.355510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355497 4486 cpu_manager.go:414] "containerMap: removing container" podUID="5a34c2c1-34c8-4938-9158-c11eb475e640" containerName="revoke-keys" Nov 04 01:32:08.355559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355511 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a34c2c1-34c8-4938-9158-c11eb475e640" containerName="revoke-keys" Nov 04 01:32:08.355559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355536 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.355559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355541 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.355656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355563 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:32:08.355656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355568 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5d3b9d-d8ea-4c19-a872-cd9b656fde6c" containerName="istio-proxy" Nov 04 01:32:08.355656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355646 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.355755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.355661 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a34c2c1-34c8-4938-9158-c11eb475e640" containerName="revoke-keys" Nov 04 01:32:08.358190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.358172 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.409517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.409484 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:32:08.409790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.409772 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:32:08.420006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.419954 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9"] Nov 04 01:32:08.420923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.420894 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.420923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.420922 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.421179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.421152 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5224ab9-4358-481d-b5ba-c7603a5266fd" containerName="istio-proxy" Nov 04 01:32:08.423703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.423677 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.437885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.434653 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxt6\" (UniqueName: \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.437885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.434744 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.437885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.434837 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.446519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.446475 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr"] Nov 04 01:32:08.484558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.484524 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9"] Nov 04 01:32:08.535736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.535698 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hdn\" (UniqueName: \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.535940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.535845 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.535940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.535925 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxt6\" (UniqueName: \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.536166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.535962 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.536166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.536082 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.536166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.536126 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.536898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.536872 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.538586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.538560 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.591542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.591514 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxt6\" (UniqueName: \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\") pod \"maas-api-revoke-sub-33d9d449-crptr\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.637468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.637386 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.637468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.637451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.637660 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.637554 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hdn\" (UniqueName: \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.638094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.638072 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.639806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.639778 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.671790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.671757 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:08.693187 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.693159 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hdn\" (UniqueName: \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\") pod \"maas-api-revoke-sub-b6eb7bf4-lwvp9\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:08.749824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:08.749792 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:09.066074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.066033 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr"] Nov 04 01:32:09.066074 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:09.066048 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7273b770_32c1_44bc_8246_1dd76167a40c.slice/crio-d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62 WatchSource:0}: Error finding container d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62: Status 404 returned error can't find the container with id d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62 Nov 04 01:32:09.182099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.182070 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9"] Nov 04 01:32:09.182434 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:09.182414 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a787480_7a17_4ed8_8270_48b80a6c23ce.slice/crio-385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06 WatchSource:0}: Error finding container 385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06: Status 404 returned error can't find the container with id 385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06 Nov 04 01:32:09.795839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.795749 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" event={"ID":"4a787480-7a17-4ed8-8270-48b80a6c23ce","Type":"ContainerStarted","Data":"8eab173a62c2a5341af5f4c68de39f576f459477af80253a1cb5f09f0647bd76"} Nov 04 01:32:09.795839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.795812 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" event={"ID":"4a787480-7a17-4ed8-8270-48b80a6c23ce","Type":"ContainerStarted","Data":"385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06"} Nov 04 01:32:09.797837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.797806 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" event={"ID":"7273b770-32c1-44bc-8246-1dd76167a40c","Type":"ContainerStarted","Data":"77e20944ea31a486e54b10d8eb2444f5cfeef2cbdd343055d1c6fc5bbad54f06"} Nov 04 01:32:09.797952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.797850 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" event={"ID":"7273b770-32c1-44bc-8246-1dd76167a40c","Type":"ContainerStarted","Data":"d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62"} Nov 04 01:32:09.920163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:09.920101 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" podStartSLOduration=1.920086662 podStartE2EDuration="1.920086662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:09.916655183 +0000 UTC m=+2098.675317344" watchObservedRunningTime="2026-11-04 01:32:09.920086662 +0000 UTC m=+2098.678749019" Nov 04 01:32:10.033855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:10.033803 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" podStartSLOduration=2.033788145 podStartE2EDuration="2.033788145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:10.031668385 +0000 UTC m=+2098.790330546" watchObservedRunningTime="2026-11-04 01:32:10.033788145 +0000 UTC m=+2098.792450306" Nov 04 01:32:10.806599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:10.806565 4486 generic.go:356] "Generic (PLEG): container finished" podID="7273b770-32c1-44bc-8246-1dd76167a40c" containerID="77e20944ea31a486e54b10d8eb2444f5cfeef2cbdd343055d1c6fc5bbad54f06" exitCode=0 Nov 04 01:32:10.807079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:10.806653 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" event={"ID":"7273b770-32c1-44bc-8246-1dd76167a40c","Type":"ContainerDied","Data":"77e20944ea31a486e54b10d8eb2444f5cfeef2cbdd343055d1c6fc5bbad54f06"} Nov 04 01:32:10.808363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:10.808339 4486 generic.go:356] "Generic (PLEG): container finished" podID="4a787480-7a17-4ed8-8270-48b80a6c23ce" containerID="8eab173a62c2a5341af5f4c68de39f576f459477af80253a1cb5f09f0647bd76" exitCode=0 Nov 04 01:32:10.808470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:10.808419 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" event={"ID":"4a787480-7a17-4ed8-8270-48b80a6c23ce","Type":"ContainerDied","Data":"8eab173a62c2a5341af5f4c68de39f576f459477af80253a1cb5f09f0647bd76"} Nov 04 01:32:11.981343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:11.981312 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:12.012248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.012216 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:12.087775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.087692 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\" (UniqueName: \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\") pod \"4a787480-7a17-4ed8-8270-48b80a6c23ce\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " Nov 04 01:32:12.087775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.087771 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\") pod \"4a787480-7a17-4ed8-8270-48b80a6c23ce\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " Nov 04 01:32:12.088016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.087892 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\") pod \"7273b770-32c1-44bc-8246-1dd76167a40c\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " Nov 04 01:32:12.088016 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.087964 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\" (UniqueName: \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\") pod \"7273b770-32c1-44bc-8246-1dd76167a40c\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " Nov 04 01:32:12.088156 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088108 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\") pod \"7273b770-32c1-44bc-8246-1dd76167a40c\" (UID: \"7273b770-32c1-44bc-8246-1dd76167a40c\") " Nov 04 01:32:12.088291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088267 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\") pod \"4a787480-7a17-4ed8-8270-48b80a6c23ce\" (UID: \"4a787480-7a17-4ed8-8270-48b80a6c23ce\") " Nov 04 01:32:12.088470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088290 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca" pod "7273b770-32c1-44bc-8246-1dd76167a40c" (UID: "7273b770-32c1-44bc-8246-1dd76167a40c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:12.088659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088637 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca" pod "4a787480-7a17-4ed8-8270-48b80a6c23ce" (UID: "4a787480-7a17-4ed8-8270-48b80a6c23ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:12.088864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088846 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.088929 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.088867 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.090542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.090516 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6" pod "7273b770-32c1-44bc-8246-1dd76167a40c" (UID: "7273b770-32c1-44bc-8246-1dd76167a40c"). InnerVolumeSpecName "kube-api-access-tzxt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:12.090634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.090548 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca" pod "4a787480-7a17-4ed8-8270-48b80a6c23ce" (UID: "4a787480-7a17-4ed8-8270-48b80a6c23ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:12.090634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.090620 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn" pod "4a787480-7a17-4ed8-8270-48b80a6c23ce" (UID: "4a787480-7a17-4ed8-8270-48b80a6c23ce"). InnerVolumeSpecName "kube-api-access-24hdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:12.090634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.090625 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca" pod "7273b770-32c1-44bc-8246-1dd76167a40c" (UID: "7273b770-32c1-44bc-8246-1dd76167a40c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:12.190603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.190565 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxt6\" (UniqueName: \"kubernetes.io/projected/7273b770-32c1-44bc-8246-1dd76167a40c-kube-api-access-tzxt6\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.190603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.190598 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7273b770-32c1-44bc-8246-1dd76167a40c-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.190603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.190609 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hdn\" (UniqueName: \"kubernetes.io/projected/4a787480-7a17-4ed8-8270-48b80a6c23ce-kube-api-access-24hdn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.190840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.190619 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a787480-7a17-4ed8-8270-48b80a6c23ce-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:12.793520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.793470 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" containerID="cri-o://cad94f8d670671aef46871b801c3c964ebff0bc491b001a7294c4dc8c78ee462" gracePeriod=25 Nov 04 01:32:12.827049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.827022 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" Nov 04 01:32:12.827049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.827035 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr" event={"ID":"7273b770-32c1-44bc-8246-1dd76167a40c","Type":"ContainerDied","Data":"d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62"} Nov 04 01:32:12.827296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.827085 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d25c2fdf4b2f402f047691d7c3c674a965afe8bfb9759626256a6c8463c72f62" Nov 04 01:32:12.828674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.828648 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" Nov 04 01:32:12.828821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.828678 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9" event={"ID":"4a787480-7a17-4ed8-8270-48b80a6c23ce","Type":"ContainerDied","Data":"385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06"} Nov 04 01:32:12.828821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.828716 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="385c20c73f2f12d7fd4eebae61b87440b34fc05133e2bcac808a5f287a0cfe06" Nov 04 01:32:12.914052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:12.913966 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" containerID="cri-o://36845891034b56c8cc6956be03240e77485747e5f38fee1746d3fe979fc804bc" gracePeriod=25 Nov 04 01:32:13.853384 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:13.853337 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:32:15.979954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:15.979918 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq"] Nov 04 01:32:16.025918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.025884 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-42958c-fhxfq"] Nov 04 01:32:16.187969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.187909 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" containerID="cri-o://95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6" gracePeriod=45 Nov 04 01:32:16.468784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.468761 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:32:16.562738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.562708 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.562922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.562761 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.562922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.562865 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.562922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.562893 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.563139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563025 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.563139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563081 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\" (UniqueName: \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.563139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563123 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\") pod \"900a9742-6ff6-4144-8d96-16b72f1065f8\" (UID: \"900a9742-6ff6-4144-8d96-16b72f1065f8\") " Nov 04 01:32:16.563296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563162 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.563415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563386 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.563728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563704 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.563831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563733 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.563831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.563749 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.565223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.565195 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:16.565327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.565269 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:16.565327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.565309 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk" pod "900a9742-6ff6-4144-8d96-16b72f1065f8" (UID: "900a9742-6ff6-4144-8d96-16b72f1065f8"). InnerVolumeSpecName "kube-api-access-76xkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:16.667064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.667025 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xkk\" (UniqueName: \"kubernetes.io/projected/900a9742-6ff6-4144-8d96-16b72f1065f8-kube-api-access-76xkk\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.667064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.667065 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900a9742-6ff6-4144-8d96-16b72f1065f8-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.667270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.667078 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900a9742-6ff6-4144-8d96-16b72f1065f8-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:16.883024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.882907 4486 generic.go:356] "Generic (PLEG): container finished" podID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerID="95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6" exitCode=0 Nov 04 01:32:16.883024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.882968 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" Nov 04 01:32:16.883024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.883006 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" event={"ID":"900a9742-6ff6-4144-8d96-16b72f1065f8","Type":"ContainerDied","Data":"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6"} Nov 04 01:32:16.883282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.883050 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr" event={"ID":"900a9742-6ff6-4144-8d96-16b72f1065f8","Type":"ContainerDied","Data":"f38da8c5800bf85e08ee51e25ab9688e9a06c6d148a030a1b43cba80e3dd0dc0"} Nov 04 01:32:16.883282 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.883071 4486 scope.go:122] "RemoveContainer" containerID="95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6" Nov 04 01:32:16.899059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.899032 4486 scope.go:122] "RemoveContainer" containerID="95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6" Nov 04 01:32:16.899405 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:16.899379 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6\": container with ID starting with 95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6 not found: ID does not exist" containerID="95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6" Nov 04 01:32:16.899493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:16.899414 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6"} err="failed to get container status \"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6\": rpc error: code = NotFound desc = could not find container \"95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6\": container with ID starting with 95f183457a1ca912ce2da825f66063905cd50ba977c498ff17461a8ef2df01b6 not found: ID does not exist" Nov 04 01:32:17.045604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.045571 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:32:17.055998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.055956 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-678aaf/distinct-w6-678aaf-kserve-859f56994d-rqxqr"] Nov 04 01:32:17.239896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.239818 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx"] Nov 04 01:32:17.240503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240485 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4a787480-7a17-4ed8-8270-48b80a6c23ce" containerName="revoke-keys" Nov 04 01:32:17.240545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240505 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a787480-7a17-4ed8-8270-48b80a6c23ce" containerName="revoke-keys" Nov 04 01:32:17.240545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240541 4486 cpu_manager.go:414] "containerMap: removing container" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" Nov 04 01:32:17.240619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240547 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" Nov 04 01:32:17.240619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240556 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7273b770-32c1-44bc-8246-1dd76167a40c" containerName="revoke-keys" Nov 04 01:32:17.240619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240562 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7273b770-32c1-44bc-8246-1dd76167a40c" containerName="revoke-keys" Nov 04 01:32:17.240714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240659 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" containerName="main" Nov 04 01:32:17.240714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240667 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7273b770-32c1-44bc-8246-1dd76167a40c" containerName="revoke-keys" Nov 04 01:32:17.240714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.240677 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a787480-7a17-4ed8-8270-48b80a6c23ce" containerName="revoke-keys" Nov 04 01:32:17.245707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.245674 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.299294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.299262 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:32:17.299294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.299266 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:32:17.306759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.306730 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx"] Nov 04 01:32:17.350407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.350375 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900a9742-6ff6-4144-8d96-16b72f1065f8" path="/var/lib/kubelet/pods/900a9742-6ff6-4144-8d96-16b72f1065f8/volumes" Nov 04 01:32:17.350824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.350807 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d1b749-01f0-44d4-8fdc-d0c50c127b02" path="/var/lib/kubelet/pods/d4d1b749-01f0-44d4-8fdc-d0c50c127b02/volumes" Nov 04 01:32:17.376312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.376281 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srthx\" (UniqueName: \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.376470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.376373 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.376470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.376403 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.477462 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.477425 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srthx\" (UniqueName: \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.477640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.477525 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.477640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.477545 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.478199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.478174 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.480017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.479996 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.534634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.534556 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srthx\" (UniqueName: \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.560241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.560210 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:17.697628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.697504 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:17.697810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.697788 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:17.955918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.955885 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:32:17.972236 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.972205 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:32:17.980107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.980082 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:32:17.985616 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:17.985583 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70532665_1e9b_4da7_872f_f5e84960d87d.slice/crio-dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f WatchSource:0}: Error finding container dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f: Status 404 returned error can't find the container with id dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f Nov 04 01:32:17.987142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:17.987062 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx"] Nov 04 01:32:18.000442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:18.000422 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:32:18.901841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:18.901808 4486 generic.go:356] "Generic (PLEG): container finished" podID="70532665-1e9b-4da7-872f-f5e84960d87d" containerID="cbc5cda7547c97c4e4c5368fdccf39ca2f8a9414573aa19661b01900fda68886" exitCode=0 Nov 04 01:32:18.902317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:18.901887 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" event={"ID":"70532665-1e9b-4da7-872f-f5e84960d87d","Type":"ContainerDied","Data":"cbc5cda7547c97c4e4c5368fdccf39ca2f8a9414573aa19661b01900fda68886"} Nov 04 01:32:18.902317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:18.901939 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" event={"ID":"70532665-1e9b-4da7-872f-f5e84960d87d","Type":"ContainerStarted","Data":"dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f"} Nov 04 01:32:20.109885 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.109858 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:20.211728 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.211638 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\" (UniqueName: \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\") pod \"70532665-1e9b-4da7-872f-f5e84960d87d\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " Nov 04 01:32:20.211877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.211815 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\") pod \"70532665-1e9b-4da7-872f-f5e84960d87d\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " Nov 04 01:32:20.212034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.212012 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\") pod \"70532665-1e9b-4da7-872f-f5e84960d87d\" (UID: \"70532665-1e9b-4da7-872f-f5e84960d87d\") " Nov 04 01:32:20.212351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.212329 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca" pod "70532665-1e9b-4da7-872f-f5e84960d87d" (UID: "70532665-1e9b-4da7-872f-f5e84960d87d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:20.212534 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.212516 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:20.213838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.213810 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca" pod "70532665-1e9b-4da7-872f-f5e84960d87d" (UID: "70532665-1e9b-4da7-872f-f5e84960d87d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:20.213938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.213864 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx" pod "70532665-1e9b-4da7-872f-f5e84960d87d" (UID: "70532665-1e9b-4da7-872f-f5e84960d87d"). InnerVolumeSpecName "kube-api-access-srthx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:20.313486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.313445 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srthx\" (UniqueName: \"kubernetes.io/projected/70532665-1e9b-4da7-872f-f5e84960d87d-kube-api-access-srthx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:20.313486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.313485 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70532665-1e9b-4da7-872f-f5e84960d87d-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:20.919744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.919714 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" Nov 04 01:32:20.919902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.919743 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx" event={"ID":"70532665-1e9b-4da7-872f-f5e84960d87d","Type":"ContainerDied","Data":"dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f"} Nov 04 01:32:20.919902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:20.919791 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbc4908f9dfae32b209b4082209ce197e1a21675948a12847cdb6a407d25727f" Nov 04 01:32:21.056880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:21.056845 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv"] Nov 04 01:32:21.060740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:21.060709 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e734d5-bz2fv"] Nov 04 01:32:21.353191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:21.353155 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbb94a6-08da-402b-93ab-e3a1d4608816" path="/var/lib/kubelet/pods/0cbb94a6-08da-402b-93ab-e3a1d4608816/volumes" Nov 04 01:32:24.162166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:24.162039 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:32:24.162637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:24.162342 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" containerID="cri-o://64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64" gracePeriod=30 Nov 04 01:32:25.204683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.204634 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7"] Nov 04 01:32:25.205678 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.205646 4486 cpu_manager.go:414] "containerMap: removing container" podUID="70532665-1e9b-4da7-872f-f5e84960d87d" containerName="revoke-keys" Nov 04 01:32:25.205678 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.205676 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70532665-1e9b-4da7-872f-f5e84960d87d" containerName="revoke-keys" Nov 04 01:32:25.205898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.205883 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="70532665-1e9b-4da7-872f-f5e84960d87d" containerName="revoke-keys" Nov 04 01:32:25.211226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.211204 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.271242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.271206 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7"] Nov 04 01:32:25.298571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.298535 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.298758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.298589 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppmv\" (UniqueName: \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.298758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.298620 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.313155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.312812 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:32:25.313155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.312892 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:32:25.400504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.400471 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.400713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.400533 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppmv\" (UniqueName: \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.400713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.400586 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.401411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.401388 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.404042 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.404016 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.463046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.460917 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppmv\" (UniqueName: \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\") pod \"maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.532007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.529175 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:25.964009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.963944 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7"] Nov 04 01:32:25.988834 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:25.988808 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" event={"ID":"5e683758-b378-4a1d-a4e4-85f88992e54f","Type":"ContainerStarted","Data":"2dbf3c715735d79d8cf4f30a45c96a7afe8bc74ddcf4a4d1dbccfeaf3b07e4df"} Nov 04 01:32:26.998738 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:26.998698 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" event={"ID":"5e683758-b378-4a1d-a4e4-85f88992e54f","Type":"ContainerStarted","Data":"b0466ad0c26e05672d5e0b5b6ba64ea74d24f8b47e0df0ec001ae8eab954216a"} Nov 04 01:32:27.118189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:27.118123 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" podStartSLOduration=2.118103351 podStartE2EDuration="2.118103351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:27.112905327 +0000 UTC m=+2115.871567488" watchObservedRunningTime="2026-11-04 01:32:27.118103351 +0000 UTC m=+2115.876765514" Nov 04 01:32:27.697515 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:27.697482 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:27.697669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:27.697488 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:28.010055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:28.009605 4486 generic.go:356] "Generic (PLEG): container finished" podID="5e683758-b378-4a1d-a4e4-85f88992e54f" containerID="b0466ad0c26e05672d5e0b5b6ba64ea74d24f8b47e0df0ec001ae8eab954216a" exitCode=0 Nov 04 01:32:28.010055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:28.009817 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" event={"ID":"5e683758-b378-4a1d-a4e4-85f88992e54f","Type":"ContainerDied","Data":"b0466ad0c26e05672d5e0b5b6ba64ea74d24f8b47e0df0ec001ae8eab954216a"} Nov 04 01:32:29.057489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.057449 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p"] Nov 04 01:32:29.124046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.121076 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh"] Nov 04 01:32:29.124046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.121166 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d40c53fb-7qc7p"] Nov 04 01:32:29.124046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.121513 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7bd7e9a-vt2xh"] Nov 04 01:32:29.331041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.327253 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:29.377781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.377739 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8439e5-a2b1-42b4-92de-0c246986a1ab" path="/var/lib/kubelet/pods/8f8439e5-a2b1-42b4-92de-0c246986a1ab/volumes" Nov 04 01:32:29.378372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.378346 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\" (UniqueName: \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\") pod \"5e683758-b378-4a1d-a4e4-85f88992e54f\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " Nov 04 01:32:29.378607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.378588 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\") pod \"5e683758-b378-4a1d-a4e4-85f88992e54f\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " Nov 04 01:32:29.378677 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.378655 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\") pod \"5e683758-b378-4a1d-a4e4-85f88992e54f\" (UID: \"5e683758-b378-4a1d-a4e4-85f88992e54f\") " Nov 04 01:32:29.379635 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.379604 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca" pod "5e683758-b378-4a1d-a4e4-85f88992e54f" (UID: "5e683758-b378-4a1d-a4e4-85f88992e54f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:29.380722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.380680 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41837e1-c491-4797-93b4-83213177db4a" path="/var/lib/kubelet/pods/e41837e1-c491-4797-93b4-83213177db4a/volumes" Nov 04 01:32:29.383918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.383875 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv" pod "5e683758-b378-4a1d-a4e4-85f88992e54f" (UID: "5e683758-b378-4a1d-a4e4-85f88992e54f"). InnerVolumeSpecName "kube-api-access-vppmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:29.384543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.384507 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca" pod "5e683758-b378-4a1d-a4e4-85f88992e54f" (UID: "5e683758-b378-4a1d-a4e4-85f88992e54f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:29.493776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.481940 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppmv\" (UniqueName: \"kubernetes.io/projected/5e683758-b378-4a1d-a4e4-85f88992e54f-kube-api-access-vppmv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.493776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.484389 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.493776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.485778 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e683758-b378-4a1d-a4e4-85f88992e54f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.524303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.524271 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:32:29.586741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.586712 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.586941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.586823 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.586941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.586907 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.586955 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587023 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587046 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:29.587095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587058 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587136 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587181 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:29.587299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587237 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587278 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:29.587299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587294 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\") pod \"f228b269-e7f1-41f0-9c3f-76e24351c64d\" (UID: \"f228b269-e7f1-41f0-9c3f-76e24351c64d\") " Nov 04 01:32:29.587954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587834 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.587954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587860 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.587954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587876 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.587954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.587887 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:29.588360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.588100 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:29.589647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.589617 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:29.589881 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.589846 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:29.590061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.589906 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "kube-api-access-8z94q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:29.590136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.590065 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo" pod "f228b269-e7f1-41f0-9c3f-76e24351c64d" (UID: "f228b269-e7f1-41f0-9c3f-76e24351c64d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:29.649483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.649447 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:32:29.670336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.670243 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:32:29.689194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689155 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.689194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689197 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.689464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689212 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z94q\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-kube-api-access-8z94q\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.689464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689228 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.689464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689243 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f228b269-e7f1-41f0-9c3f-76e24351c64d-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.689464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:29.689256 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f228b269-e7f1-41f0-9c3f-76e24351c64d-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:30.029182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.029096 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" Nov 04 01:32:30.029182 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.029122 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7" event={"ID":"5e683758-b378-4a1d-a4e4-85f88992e54f","Type":"ContainerDied","Data":"2dbf3c715735d79d8cf4f30a45c96a7afe8bc74ddcf4a4d1dbccfeaf3b07e4df"} Nov 04 01:32:30.029428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.029183 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dbf3c715735d79d8cf4f30a45c96a7afe8bc74ddcf4a4d1dbccfeaf3b07e4df" Nov 04 01:32:30.031292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.031267 4486 generic.go:356] "Generic (PLEG): container finished" podID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerID="64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64" exitCode=0 Nov 04 01:32:30.031418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.031322 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" Nov 04 01:32:30.031418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.031358 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerDied","Data":"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64"} Nov 04 01:32:30.031418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.031412 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686" event={"ID":"f228b269-e7f1-41f0-9c3f-76e24351c64d","Type":"ContainerDied","Data":"dac0ea2d74499617e14f2759d4b7caa8050e87e5d03dc31a5b192f0f3f0e6275"} Nov 04 01:32:30.031605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.031442 4486 scope.go:122] "RemoveContainer" containerID="64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64" Nov 04 01:32:30.044957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.044938 4486 scope.go:122] "RemoveContainer" containerID="78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925" Nov 04 01:32:30.057298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.057270 4486 scope.go:122] "RemoveContainer" containerID="64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64" Nov 04 01:32:30.057597 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:30.057576 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64\": container with ID starting with 64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64 not found: ID does not exist" containerID="64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64" Nov 04 01:32:30.057956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.057609 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64"} err="failed to get container status \"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64\": rpc error: code = NotFound desc = could not find container \"64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64\": container with ID starting with 64118168e7ad001522aa053e63926c47e9d51cd88416ad4fc9fa452e6f426a64 not found: ID does not exist" Nov 04 01:32:30.057956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.057627 4486 scope.go:122] "RemoveContainer" containerID="78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925" Nov 04 01:32:30.057956 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:30.057860 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925\": container with ID starting with 78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925 not found: ID does not exist" containerID="78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925" Nov 04 01:32:30.057956 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.057878 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925"} err="failed to get container status \"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925\": rpc error: code = NotFound desc = could not find container \"78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925\": container with ID starting with 78540ee79bc84655bf3f418289a87934db5d84527a5169c52002a9a8fed21925 not found: ID does not exist" Nov 04 01:32:30.210272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.210235 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:32:30.220351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:30.220323 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-42958c-openshift-default-77ccb4bd5f-cm686"] Nov 04 01:32:31.354189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:31.354148 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" path="/var/lib/kubelet/pods/f228b269-e7f1-41f0-9c3f-76e24351c64d/volumes" Nov 04 01:32:33.068439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.068379 4486 generic.go:356] "Generic (PLEG): container finished" podID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerID="2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f" exitCode=0 Nov 04 01:32:33.069011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.068522 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerDied","Data":"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f"} Nov 04 01:32:33.069102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.069079 4486 scope.go:122] "RemoveContainer" containerID="2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f" Nov 04 01:32:33.072414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.072381 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:32:33.072547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.072470 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:32:33.072547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.072539 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:32:33.076441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.076341 4486 generic.go:356] "Generic (PLEG): container finished" podID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerID="36845891034b56c8cc6956be03240e77485747e5f38fee1746d3fe979fc804bc" exitCode=0 Nov 04 01:32:33.076576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.076449 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" event={"ID":"026e0e82-97c5-4662-bb88-ec0937b8f5f0","Type":"ContainerDied","Data":"36845891034b56c8cc6956be03240e77485747e5f38fee1746d3fe979fc804bc"} Nov 04 01:32:33.078302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.078280 4486 generic.go:356] "Generic (PLEG): container finished" podID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerID="cad94f8d670671aef46871b801c3c964ebff0bc491b001a7294c4dc8c78ee462" exitCode=0 Nov 04 01:32:33.078513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.078494 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" event={"ID":"c5526139-52c9-440d-bc9f-a3bb1fcad224","Type":"ContainerDied","Data":"cad94f8d670671aef46871b801c3c964ebff0bc491b001a7294c4dc8c78ee462"} Nov 04 01:32:33.109637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.109607 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:32:33.200799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.200772 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:32:33.258431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.258140 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\" (UniqueName: \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\") pod \"c5526139-52c9-440d-bc9f-a3bb1fcad224\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " Nov 04 01:32:33.258629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.258479 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\") pod \"c5526139-52c9-440d-bc9f-a3bb1fcad224\" (UID: \"c5526139-52c9-440d-bc9f-a3bb1fcad224\") " Nov 04 01:32:33.259900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.259649 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume" pod "c5526139-52c9-440d-bc9f-a3bb1fcad224" (UID: "c5526139-52c9-440d-bc9f-a3bb1fcad224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:33.264360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.261279 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6" pod "c5526139-52c9-440d-bc9f-a3bb1fcad224" (UID: "c5526139-52c9-440d-bc9f-a3bb1fcad224"). InnerVolumeSpecName "kube-api-access-r4wg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:33.359852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.359709 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\" (UniqueName: \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\") pod \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " Nov 04 01:32:33.359852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.359808 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\") pod \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\" (UID: \"026e0e82-97c5-4662-bb88-ec0937b8f5f0\") " Nov 04 01:32:33.360350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.360316 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5526139-52c9-440d-bc9f-a3bb1fcad224-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:33.360432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.360361 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wg6\" (UniqueName: \"kubernetes.io/projected/c5526139-52c9-440d-bc9f-a3bb1fcad224-kube-api-access-r4wg6\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:33.360735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.360710 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume" pod "026e0e82-97c5-4662-bb88-ec0937b8f5f0" (UID: "026e0e82-97c5-4662-bb88-ec0937b8f5f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:33.363641 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.363614 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9" pod "026e0e82-97c5-4662-bb88-ec0937b8f5f0" (UID: "026e0e82-97c5-4662-bb88-ec0937b8f5f0"). InnerVolumeSpecName "kube-api-access-sstn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:33.462618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.462191 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstn9\" (UniqueName: \"kubernetes.io/projected/026e0e82-97c5-4662-bb88-ec0937b8f5f0-kube-api-access-sstn9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:33.462618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:33.462226 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026e0e82-97c5-4662-bb88-ec0937b8f5f0-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:34.088962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.088877 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" Nov 04 01:32:34.088962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.088896 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9" event={"ID":"026e0e82-97c5-4662-bb88-ec0937b8f5f0","Type":"ContainerDied","Data":"954f9ce68e106e7bc8fc3d9aed5fe6e8698cdadff62d420b828b29f5b052f11b"} Nov 04 01:32:34.089488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.088962 4486 scope.go:122] "RemoveContainer" containerID="36845891034b56c8cc6956be03240e77485747e5f38fee1746d3fe979fc804bc" Nov 04 01:32:34.090692 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.090670 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" Nov 04 01:32:34.090807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.090689 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww" event={"ID":"c5526139-52c9-440d-bc9f-a3bb1fcad224","Type":"ContainerDied","Data":"8b53902499db0892c0a1ed3284d66dc8af74f048ef71d6217580a5887012c11d"} Nov 04 01:32:34.093526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.093473 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerStarted","Data":"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09"} Nov 04 01:32:34.100933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.100917 4486 scope.go:122] "RemoveContainer" containerID="cad94f8d670671aef46871b801c3c964ebff0bc491b001a7294c4dc8c78ee462" Nov 04 01:32:34.365083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.353333 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:32:34.365271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.365248 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:34.365341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.365304 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:34.401647 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.401617 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:32:34.412924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.412882 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-678aaf-55476f8864-nrmww"] Nov 04 01:32:34.561009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.560019 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:32:34.566490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:34.566458 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-678aaf-84b8f68c75-dlkx9"] Nov 04 01:32:35.360355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:35.360316 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:35.360931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:35.360898 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:35.366170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:35.366140 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" path="/var/lib/kubelet/pods/026e0e82-97c5-4662-bb88-ec0937b8f5f0/volumes" Nov 04 01:32:35.366767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:35.366746 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" path="/var/lib/kubelet/pods/c5526139-52c9-440d-bc9f-a3bb1fcad224/volumes" Nov 04 01:32:36.353453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:36.353413 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:36.353676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:36.353482 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:36.761922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:36.761825 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:32:36.886324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:36.886293 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:32:37.364007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:37.356939 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:37.364007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:37.357018 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:38.369009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:38.367302 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:38.369009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:38.367377 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:39.194235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:39.194204 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:32:39.352402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:39.352366 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Nov 04 01:32:39.352606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:39.352417 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Nov 04 01:32:40.089162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:40.089126 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx"] Nov 04 01:32:40.092663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:40.092634 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-5778ad-mvvfx"] Nov 04 01:32:40.357792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:40.357722 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:32:40.358701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:40.358678 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:32:41.352699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:41.352665 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70532665-1e9b-4da7-872f-f5e84960d87d" path="/var/lib/kubelet/pods/70532665-1e9b-4da7-872f-f5e84960d87d/volumes" Nov 04 01:32:41.762583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:41.762493 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" containerID="cri-o://4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe" gracePeriod=25 Nov 04 01:32:41.887481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:41.887436 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" containerID="cri-o://42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864" gracePeriod=25 Nov 04 01:32:42.666051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:42.666014 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:42.666051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:42.666032 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:43.831325 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:43.831287 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:32:44.679324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.679276 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" containerID="cri-o://e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b" gracePeriod=45 Nov 04 01:32:44.693411 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.693382 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" containerID="cri-o://c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567" gracePeriod=45 Nov 04 01:32:44.751164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.751131 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh"] Nov 04 01:32:44.752013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.751957 4486 cpu_manager.go:414] "containerMap: removing container" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" Nov 04 01:32:44.752013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.751995 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752034 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752041 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752055 4486 cpu_manager.go:414] "containerMap: removing container" podUID="5e683758-b378-4a1d-a4e4-85f88992e54f" containerName="revoke-keys" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752068 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e683758-b378-4a1d-a4e4-85f88992e54f" containerName="revoke-keys" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752079 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" Nov 04 01:32:44.752190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752085 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" Nov 04 01:32:44.752403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752226 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.752403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752237 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="026e0e82-97c5-4662-bb88-ec0937b8f5f0" containerName="payload-pre-processing" Nov 04 01:32:44.752403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752252 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e683758-b378-4a1d-a4e4-85f88992e54f" containerName="revoke-keys" Nov 04 01:32:44.752403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.752261 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5526139-52c9-440d-bc9f-a3bb1fcad224" containerName="payload-processing" Nov 04 01:32:44.754843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.754822 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.813058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.813014 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr"] Nov 04 01:32:44.813884 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.813860 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.814091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.814075 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.814402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.814388 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f228b269-e7f1-41f0-9c3f-76e24351c64d" containerName="istio-proxy" Nov 04 01:32:44.818222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.817625 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:44.860166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.859163 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:32:44.860166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.859735 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.860166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.859825 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhvt\" (UniqueName: \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.860166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.859864 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.861167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.860451 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:32:44.863049 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.863024 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh"] Nov 04 01:32:44.888210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.888174 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr"] Nov 04 01:32:44.961748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.961639 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.962536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.962418 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhvt\" (UniqueName: \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.962536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.962475 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.962536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.962527 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:44.962933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.962594 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:44.962933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.962643 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27nw\" (UniqueName: \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:44.963489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.963464 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:44.966092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:44.966060 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:45.018242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.018215 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:32:45.021113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.021092 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:32:45.023489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.023471 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhvt\" (UniqueName: \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\") pod \"maas-api-revoke-sub-cdb77332-qdrrh\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:45.063629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.063598 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.063739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.063660 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.063739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.063707 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27nw\" (UniqueName: \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.064343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.064323 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.065848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.065830 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.089029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.089007 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:45.134323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.134289 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27nw\" (UniqueName: \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\") pod \"maas-api-revoke-sub-0a4febda-gn6rr\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.150204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.150100 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:45.164590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164558 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.164700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164643 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\" (UniqueName: \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.164778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164702 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\" (UniqueName: \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.164778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164758 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.164889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164800 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.164889 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164863 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.165010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.164898 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.165010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165000 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.165126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165038 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.165126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165076 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.165232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165161 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\" (UniqueName: \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\") pod \"8113b2a6-4fed-41d8-be88-d04cac9a8569\" (UID: \"8113b2a6-4fed-41d8-be88-d04cac9a8569\") " Nov 04 01:32:45.165232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165191 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\") pod \"0a7f46ba-88fe-4622-b91a-07c42836b76a\" (UID: \"0a7f46ba-88fe-4622-b91a-07c42836b76a\") " Nov 04 01:32:45.165331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165240 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.165482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165457 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.166370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165741 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.166370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165803 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.166370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.165823 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.171063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.170489 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.171063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.170851 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.171512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.171485 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.174768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.174742 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:45.178495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.178460 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:45.179440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.179398 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "kube-api-access-46gms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:45.180747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.180719 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm" pod "8113b2a6-4fed-41d8-be88-d04cac9a8569" (UID: "8113b2a6-4fed-41d8-be88-d04cac9a8569"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.181142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.181118 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:45.181976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.181949 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj" pod "0a7f46ba-88fe-4622-b91a-07c42836b76a" (UID: "0a7f46ba-88fe-4622-b91a-07c42836b76a"). InnerVolumeSpecName "kube-api-access-2xktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:45.211249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.206466 4486 generic.go:356] "Generic (PLEG): container finished" podID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerID="e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b" exitCode=0 Nov 04 01:32:45.211249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.206574 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" event={"ID":"0a7f46ba-88fe-4622-b91a-07c42836b76a","Type":"ContainerDied","Data":"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b"} Nov 04 01:32:45.211249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.206614 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" event={"ID":"0a7f46ba-88fe-4622-b91a-07c42836b76a","Type":"ContainerDied","Data":"74572a0d93ae5bbf704fff3c58436ea9f9d8de411d45b07b52731ebe67a82c9a"} Nov 04 01:32:45.211249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.206637 4486 scope.go:122] "RemoveContainer" containerID="e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b" Nov 04 01:32:45.211249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.206803 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl" Nov 04 01:32:45.212871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.212295 4486 generic.go:356] "Generic (PLEG): container finished" podID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerID="c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567" exitCode=0 Nov 04 01:32:45.212871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.212408 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" event={"ID":"8113b2a6-4fed-41d8-be88-d04cac9a8569","Type":"ContainerDied","Data":"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567"} Nov 04 01:32:45.212871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.212444 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" event={"ID":"8113b2a6-4fed-41d8-be88-d04cac9a8569","Type":"ContainerDied","Data":"c3d80bff11f436afd4b1d631032fa186a0652db8ab929861c8099c1b8b3a6fe0"} Nov 04 01:32:45.212871 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.212508 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb" Nov 04 01:32:45.233221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.226327 4486 scope.go:122] "RemoveContainer" containerID="e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b" Nov 04 01:32:45.233221 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:45.226597 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b\": container with ID starting with e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b not found: ID does not exist" containerID="e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b" Nov 04 01:32:45.233221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.226628 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b"} err="failed to get container status \"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b\": rpc error: code = NotFound desc = could not find container \"e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b\": container with ID starting with e25a9244860c841df28334147dcb9f505931b8abb79fb7c6767dc8344295de5b not found: ID does not exist" Nov 04 01:32:45.233221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.226652 4486 scope.go:122] "RemoveContainer" containerID="c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567" Nov 04 01:32:45.242165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.242079 4486 scope.go:122] "RemoveContainer" containerID="c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567" Nov 04 01:32:45.242808 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:32:45.242778 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567\": container with ID starting with c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567 not found: ID does not exist" containerID="c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567" Nov 04 01:32:45.242938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.242903 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567"} err="failed to get container status \"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567\": rpc error: code = NotFound desc = could not find container \"c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567\": container with ID starting with c97403aa1091fc5154b78f697305b87632ab16c216bb73061fe19234c2cef567 not found: ID does not exist" Nov 04 01:32:45.266958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.266928 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7f46ba-88fe-4622-b91a-07c42836b76a-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.266966 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267017 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8113b2a6-4fed-41d8-be88-d04cac9a8569-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267033 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267047 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267060 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xktj\" (UniqueName: \"kubernetes.io/projected/0a7f46ba-88fe-4622-b91a-07c42836b76a-kube-api-access-2xktj\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267075 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46gms\" (UniqueName: \"kubernetes.io/projected/8113b2a6-4fed-41d8-be88-d04cac9a8569-kube-api-access-46gms\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267089 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267102 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8113b2a6-4fed-41d8-be88-d04cac9a8569-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.267237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.267115 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7f46ba-88fe-4622-b91a-07c42836b76a-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:45.416927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.416896 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:32:45.420778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.420749 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/premium-w0-e993d7-kserve-8964c87f-ql2zb"] Nov 04 01:32:45.472845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.472817 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh"] Nov 04 01:32:45.473496 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:45.473468 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd3fdd7_9484_4527_b713_a043371a9da9.slice/crio-19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05 WatchSource:0}: Error finding container 19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05: Status 404 returned error can't find the container with id 19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05 Nov 04 01:32:45.533571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.533541 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:32:45.540610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.540586 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e993d7/distinct-w0-e993d7-kserve-774f76c846-v2frl"] Nov 04 01:32:45.567306 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:45.567269 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bb5d23_ea17_4913_ac6d_d5737f32b038.slice/crio-4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9 WatchSource:0}: Error finding container 4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9: Status 404 returned error can't find the container with id 4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9 Nov 04 01:32:45.567419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:45.567318 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr"] Nov 04 01:32:46.222263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.222225 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" event={"ID":"9dd3fdd7-9484-4527-b713-a043371a9da9","Type":"ContainerStarted","Data":"7a513eb47eef572ebad7c446e31c3ade0b02820d3d6c3a4a59a59966977fc284"} Nov 04 01:32:46.222716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.222285 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" event={"ID":"9dd3fdd7-9484-4527-b713-a043371a9da9","Type":"ContainerStarted","Data":"19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05"} Nov 04 01:32:46.224027 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.224003 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" event={"ID":"38bb5d23-ea17-4913-ac6d-d5737f32b038","Type":"ContainerStarted","Data":"379b33f32e30ced139e73dc0ae1a29edb15cf45c001e6a918310f4fa15e2894d"} Nov 04 01:32:46.224146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.224050 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" event={"ID":"38bb5d23-ea17-4913-ac6d-d5737f32b038","Type":"ContainerStarted","Data":"4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9"} Nov 04 01:32:46.336822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.336743 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" podStartSLOduration=2.336728543 podStartE2EDuration="2.336728543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:46.332671375 +0000 UTC m=+2135.091333538" watchObservedRunningTime="2026-11-04 01:32:46.336728543 +0000 UTC m=+2135.095390704" Nov 04 01:32:46.450011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:46.447185 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" podStartSLOduration=2.447166858 podStartE2EDuration="2.447166858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:46.443611375 +0000 UTC m=+2135.202273535" watchObservedRunningTime="2026-11-04 01:32:46.447166858 +0000 UTC m=+2135.205829020" Nov 04 01:32:47.236019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.235959 4486 generic.go:356] "Generic (PLEG): container finished" podID="9dd3fdd7-9484-4527-b713-a043371a9da9" containerID="7a513eb47eef572ebad7c446e31c3ade0b02820d3d6c3a4a59a59966977fc284" exitCode=0 Nov 04 01:32:47.236510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.236047 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" event={"ID":"9dd3fdd7-9484-4527-b713-a043371a9da9","Type":"ContainerDied","Data":"7a513eb47eef572ebad7c446e31c3ade0b02820d3d6c3a4a59a59966977fc284"} Nov 04 01:32:47.238002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.237959 4486 generic.go:356] "Generic (PLEG): container finished" podID="38bb5d23-ea17-4913-ac6d-d5737f32b038" containerID="379b33f32e30ced139e73dc0ae1a29edb15cf45c001e6a918310f4fa15e2894d" exitCode=0 Nov 04 01:32:47.238165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.238097 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" event={"ID":"38bb5d23-ea17-4913-ac6d-d5737f32b038","Type":"ContainerDied","Data":"379b33f32e30ced139e73dc0ae1a29edb15cf45c001e6a918310f4fa15e2894d"} Nov 04 01:32:47.349417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.349386 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" path="/var/lib/kubelet/pods/0a7f46ba-88fe-4622-b91a-07c42836b76a/volumes" Nov 04 01:32:47.349857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:47.349843 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" path="/var/lib/kubelet/pods/8113b2a6-4fed-41d8-be88-d04cac9a8569/volumes" Nov 04 01:32:48.366329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.366293 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz"] Nov 04 01:32:48.366963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.366945 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" Nov 04 01:32:48.367046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.366964 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" Nov 04 01:32:48.367046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.367003 4486 cpu_manager.go:414] "containerMap: removing container" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" Nov 04 01:32:48.367046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.367012 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" Nov 04 01:32:48.367224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.367164 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7f46ba-88fe-4622-b91a-07c42836b76a" containerName="main" Nov 04 01:32:48.367224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.367179 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="8113b2a6-4fed-41d8-be88-d04cac9a8569" containerName="main" Nov 04 01:32:48.370216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.370194 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.423662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.423630 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:48.431454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.431428 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz"] Nov 04 01:32:48.446702 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.446676 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:48.509224 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.509124 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdf6p\" (UniqueName: \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.509379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.509274 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.509463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.509396 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.610508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610476 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\" (UniqueName: \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\") pod \"38bb5d23-ea17-4913-ac6d-d5737f32b038\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " Nov 04 01:32:48.610688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610567 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\") pod \"38bb5d23-ea17-4913-ac6d-d5737f32b038\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " Nov 04 01:32:48.610778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610702 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\") pod \"9dd3fdd7-9484-4527-b713-a043371a9da9\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " Nov 04 01:32:48.610778 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610738 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\" (UniqueName: \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\") pod \"9dd3fdd7-9484-4527-b713-a043371a9da9\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " Nov 04 01:32:48.610896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610827 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\") pod \"38bb5d23-ea17-4913-ac6d-d5737f32b038\" (UID: \"38bb5d23-ea17-4913-ac6d-d5737f32b038\") " Nov 04 01:32:48.610896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.610858 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\") pod \"9dd3fdd7-9484-4527-b713-a043371a9da9\" (UID: \"9dd3fdd7-9484-4527-b713-a043371a9da9\") " Nov 04 01:32:48.611098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.611078 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.611289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.611273 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdf6p\" (UniqueName: \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.611391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.611307 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.612091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.611818 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca" pod "9dd3fdd7-9484-4527-b713-a043371a9da9" (UID: "9dd3fdd7-9484-4527-b713-a043371a9da9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:48.612769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.612740 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.613298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.613258 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw" pod "38bb5d23-ea17-4913-ac6d-d5737f32b038" (UID: "38bb5d23-ea17-4913-ac6d-d5737f32b038"). InnerVolumeSpecName "kube-api-access-g27nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:48.613439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.613412 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca" pod "38bb5d23-ea17-4913-ac6d-d5737f32b038" (UID: "38bb5d23-ea17-4913-ac6d-d5737f32b038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:48.614157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.614133 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt" pod "9dd3fdd7-9484-4527-b713-a043371a9da9" (UID: "9dd3fdd7-9484-4527-b713-a043371a9da9"). InnerVolumeSpecName "kube-api-access-9lhvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:48.615296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.615270 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.615477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.615456 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca" pod "9dd3fdd7-9484-4527-b713-a043371a9da9" (UID: "9dd3fdd7-9484-4527-b713-a043371a9da9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:48.615572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.615547 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca" pod "38bb5d23-ea17-4913-ac6d-d5737f32b038" (UID: "38bb5d23-ea17-4913-ac6d-d5737f32b038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:48.666723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.666696 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdf6p\" (UniqueName: \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.688586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.688561 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:48.712974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.712938 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:48.713079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.712976 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27nw\" (UniqueName: \"kubernetes.io/projected/38bb5d23-ea17-4913-ac6d-d5737f32b038-kube-api-access-g27nw\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:48.713079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.713022 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:48.713079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.713031 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd3fdd7-9484-4527-b713-a043371a9da9-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:48.713079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.713041 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhvt\" (UniqueName: \"kubernetes.io/projected/9dd3fdd7-9484-4527-b713-a043371a9da9-kube-api-access-9lhvt\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:48.713079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:48.713050 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38bb5d23-ea17-4913-ac6d-d5737f32b038-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:49.074015 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:32:49.073970 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda198c8d3_175f_4d54_b31e_c401e3043879.slice/crio-8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60 WatchSource:0}: Error finding container 8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60: Status 404 returned error can't find the container with id 8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60 Nov 04 01:32:49.074170 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.074072 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz"] Nov 04 01:32:49.254649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.254615 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" Nov 04 01:32:49.254815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.254656 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh" event={"ID":"9dd3fdd7-9484-4527-b713-a043371a9da9","Type":"ContainerDied","Data":"19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05"} Nov 04 01:32:49.254815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.254699 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c74a4a934318513eb43f59188c716b8a9176a94b4e2639fcf6102a69a05d05" Nov 04 01:32:49.256547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.256524 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" Nov 04 01:32:49.256674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.256573 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr" event={"ID":"38bb5d23-ea17-4913-ac6d-d5737f32b038","Type":"ContainerDied","Data":"4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9"} Nov 04 01:32:49.256674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.256622 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0ea2dfcc309dd8c6954d909664b038fb07293770010b124b172a6ea770d2b9" Nov 04 01:32:49.259551 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:49.259526 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" event={"ID":"a198c8d3-175f-4d54-b31e-c401e3043879","Type":"ContainerStarted","Data":"8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60"} Nov 04 01:32:50.269605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:50.269515 4486 generic.go:356] "Generic (PLEG): container finished" podID="a198c8d3-175f-4d54-b31e-c401e3043879" containerID="ef1d6ea6e9a75bf04af3e6164ce15a479b3d766ebc1739147d610aac084daa89" exitCode=0 Nov 04 01:32:50.269605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:50.269565 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" event={"ID":"a198c8d3-175f-4d54-b31e-c401e3043879","Type":"ContainerDied","Data":"ef1d6ea6e9a75bf04af3e6164ce15a479b3d766ebc1739147d610aac084daa89"} Nov 04 01:32:51.472788 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.466734 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:51.552806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.552725 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\") pod \"a198c8d3-175f-4d54-b31e-c401e3043879\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " Nov 04 01:32:51.552963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.552943 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\") pod \"a198c8d3-175f-4d54-b31e-c401e3043879\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " Nov 04 01:32:51.553033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.552995 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\" (UniqueName: \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\") pod \"a198c8d3-175f-4d54-b31e-c401e3043879\" (UID: \"a198c8d3-175f-4d54-b31e-c401e3043879\") " Nov 04 01:32:51.553362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.553327 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca" pod "a198c8d3-175f-4d54-b31e-c401e3043879" (UID: "a198c8d3-175f-4d54-b31e-c401e3043879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:51.553566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.553549 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.555233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.555209 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p" pod "a198c8d3-175f-4d54-b31e-c401e3043879" (UID: "a198c8d3-175f-4d54-b31e-c401e3043879"). InnerVolumeSpecName "kube-api-access-qdf6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:51.555370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.555354 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca" pod "a198c8d3-175f-4d54-b31e-c401e3043879" (UID: "a198c8d3-175f-4d54-b31e-c401e3043879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:51.654566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.654530 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdf6p\" (UniqueName: \"kubernetes.io/projected/a198c8d3-175f-4d54-b31e-c401e3043879-kube-api-access-qdf6p\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.654566 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:51.654562 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a198c8d3-175f-4d54-b31e-c401e3043879-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:52.286239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.286205 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" Nov 04 01:32:52.286239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.286231 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz" event={"ID":"a198c8d3-175f-4d54-b31e-c401e3043879","Type":"ContainerDied","Data":"8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60"} Nov 04 01:32:52.286470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.286278 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e8ba49887cbfa57e214e3d9acc8f9e65e4f2647f8a8ee2fd77710fcd159df60" Nov 04 01:32:52.666369 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.666335 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:52.666852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.666444 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:32:52.765324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.765287 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz"] Nov 04 01:32:52.768953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:52.768926 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-678aaf-88nlz"] Nov 04 01:32:53.351336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:32:53.351291 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a34c2c1-34c8-4938-9158-c11eb475e640" path="/var/lib/kubelet/pods/5a34c2c1-34c8-4938-9158-c11eb475e640/volumes" Nov 04 01:33:01.501406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.501359 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694"] Nov 04 01:33:01.502061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502042 4486 cpu_manager.go:414] "containerMap: removing container" podUID="9dd3fdd7-9484-4527-b713-a043371a9da9" containerName="revoke-keys" Nov 04 01:33:01.502061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502062 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd3fdd7-9484-4527-b713-a043371a9da9" containerName="revoke-keys" Nov 04 01:33:01.502205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502079 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a198c8d3-175f-4d54-b31e-c401e3043879" containerName="revoke-keys" Nov 04 01:33:01.502205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502085 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a198c8d3-175f-4d54-b31e-c401e3043879" containerName="revoke-keys" Nov 04 01:33:01.502205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502102 4486 cpu_manager.go:414] "containerMap: removing container" podUID="38bb5d23-ea17-4913-ac6d-d5737f32b038" containerName="revoke-keys" Nov 04 01:33:01.502205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502107 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38bb5d23-ea17-4913-ac6d-d5737f32b038" containerName="revoke-keys" Nov 04 01:33:01.502205 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502203 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd3fdd7-9484-4527-b713-a043371a9da9" containerName="revoke-keys" Nov 04 01:33:01.502386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502216 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a198c8d3-175f-4d54-b31e-c401e3043879" containerName="revoke-keys" Nov 04 01:33:01.502386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.502229 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="38bb5d23-ea17-4913-ac6d-d5737f32b038" containerName="revoke-keys" Nov 04 01:33:01.505172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.505156 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.614301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.604304 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:33:01.614301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.604378 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:01.637053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.632957 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrbf\" (UniqueName: \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.637371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.635678 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694"] Nov 04 01:33:01.637642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.637369 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.637850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.637832 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.742782 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.742734 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrbf\" (UniqueName: \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.757210 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.757116 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.762414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.762385 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.773581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.762264 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.773776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.769668 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:01.838714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:01.838677 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrbf\" (UniqueName: \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:02.120774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.120708 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:02.200694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.200654 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:33:02.253179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.253149 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:33:02.301058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.300927 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\" (UniqueName: \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\") pod \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " Nov 04 01:33:02.301058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.300977 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\" (UniqueName: \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\") pod \"36ae019e-b51e-4576-9487-4ebc38c89d1a\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " Nov 04 01:33:02.301300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301116 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\") pod \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\" (UID: \"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0\") " Nov 04 01:33:02.301300 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301146 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\") pod \"36ae019e-b51e-4576-9487-4ebc38c89d1a\" (UID: \"36ae019e-b51e-4576-9487-4ebc38c89d1a\") " Nov 04 01:33:02.301597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301555 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume" pod "0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" (UID: "0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:02.301597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301583 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume" pod "36ae019e-b51e-4576-9487-4ebc38c89d1a" (UID: "36ae019e-b51e-4576-9487-4ebc38c89d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:02.301854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301829 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.301913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.301861 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36ae019e-b51e-4576-9487-4ebc38c89d1a-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.303439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.303416 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz" pod "36ae019e-b51e-4576-9487-4ebc38c89d1a" (UID: "36ae019e-b51e-4576-9487-4ebc38c89d1a"). InnerVolumeSpecName "kube-api-access-rblxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:02.303547 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.303490 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf" pod "0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" (UID: "0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0"). InnerVolumeSpecName "kube-api-access-sdfbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:02.369088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.369049 4486 generic.go:356] "Generic (PLEG): container finished" podID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerID="4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe" exitCode=0 Nov 04 01:33:02.369088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.369090 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" Nov 04 01:33:02.369304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.369133 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" event={"ID":"36ae019e-b51e-4576-9487-4ebc38c89d1a","Type":"ContainerDied","Data":"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe"} Nov 04 01:33:02.369304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.369195 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52" event={"ID":"36ae019e-b51e-4576-9487-4ebc38c89d1a","Type":"ContainerDied","Data":"e6945417bb20eee3fdb5fbc8f6eed37689ab9a1bc4fd1886da2d025f06538c5e"} Nov 04 01:33:02.369304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.369221 4486 scope.go:122] "RemoveContainer" containerID="4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe" Nov 04 01:33:02.371081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.371054 4486 generic.go:356] "Generic (PLEG): container finished" podID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerID="42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864" exitCode=0 Nov 04 01:33:02.371221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.371091 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" Nov 04 01:33:02.371221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.371155 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" event={"ID":"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0","Type":"ContainerDied","Data":"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864"} Nov 04 01:33:02.371221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.371199 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz" event={"ID":"0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0","Type":"ContainerDied","Data":"6e0e92e3cec22cf5f7618800dfeb3d294e4c9bf4cd94690bff801573d16f1c89"} Nov 04 01:33:02.382790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.382768 4486 scope.go:122] "RemoveContainer" containerID="4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe" Nov 04 01:33:02.383326 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:02.383293 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe\": container with ID starting with 4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe not found: ID does not exist" containerID="4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe" Nov 04 01:33:02.383407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.383336 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe"} err="failed to get container status \"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe\": rpc error: code = NotFound desc = could not find container \"4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe\": container with ID starting with 4b712c16a071b1649506482b74b8462080e8208aa56873fa7055c74348d0b5fe not found: ID does not exist" Nov 04 01:33:02.383407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.383360 4486 scope.go:122] "RemoveContainer" containerID="42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864" Nov 04 01:33:02.396413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.396390 4486 scope.go:122] "RemoveContainer" containerID="42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864" Nov 04 01:33:02.396796 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:02.396757 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864\": container with ID starting with 42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864 not found: ID does not exist" containerID="42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864" Nov 04 01:33:02.396861 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.396798 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864"} err="failed to get container status \"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864\": rpc error: code = NotFound desc = could not find container \"42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864\": container with ID starting with 42d4ee9e69d2ba6593d6470f760a2e5700a4693caad0be3de815a2811f49d864 not found: ID does not exist" Nov 04 01:33:02.403671 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.403649 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdfbf\" (UniqueName: \"kubernetes.io/projected/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0-kube-api-access-sdfbf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.403743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.403676 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblxz\" (UniqueName: \"kubernetes.io/projected/36ae019e-b51e-4576-9487-4ebc38c89d1a-kube-api-access-rblxz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:02.535819 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.535786 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694"] Nov 04 01:33:02.536352 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:33:02.536329 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ca0614_d286_4cda_b2ff_30f207637882.slice/crio-cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0 WatchSource:0}: Error finding container cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0: Status 404 returned error can't find the container with id cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0 Nov 04 01:33:02.540276 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.540247 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:33:02.543922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.543893 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e993d7-777f89db68-7rx52"] Nov 04 01:33:02.648288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.648247 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:33:02.654529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:02.654498 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e993d7-89894d467-7rwvz"] Nov 04 01:33:03.351196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:03.351161 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" path="/var/lib/kubelet/pods/0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0/volumes" Nov 04 01:33:03.351605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:03.351591 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" path="/var/lib/kubelet/pods/36ae019e-b51e-4576-9487-4ebc38c89d1a/volumes" Nov 04 01:33:03.382418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:03.382379 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" event={"ID":"95ca0614-d286-4cda-b2ff-30f207637882","Type":"ContainerStarted","Data":"d1d503e0cc463791302f1b348ce283faee08a9062fca09417f0e3e817eb7c563"} Nov 04 01:33:03.382602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:03.382443 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" event={"ID":"95ca0614-d286-4cda-b2ff-30f207637882","Type":"ContainerStarted","Data":"cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0"} Nov 04 01:33:03.505588 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:03.505529 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" podStartSLOduration=2.505515664 podStartE2EDuration="2.505515664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:03.504950445 +0000 UTC m=+2152.263612607" watchObservedRunningTime="2026-11-04 01:33:03.505515664 +0000 UTC m=+2152.264177862" Nov 04 01:33:04.394500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:04.394464 4486 generic.go:356] "Generic (PLEG): container finished" podID="95ca0614-d286-4cda-b2ff-30f207637882" containerID="d1d503e0cc463791302f1b348ce283faee08a9062fca09417f0e3e817eb7c563" exitCode=0 Nov 04 01:33:04.394938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:04.394553 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" event={"ID":"95ca0614-d286-4cda-b2ff-30f207637882","Type":"ContainerDied","Data":"d1d503e0cc463791302f1b348ce283faee08a9062fca09417f0e3e817eb7c563"} Nov 04 01:33:05.596734 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.596705 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:05.751158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.751034 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\" (UniqueName: \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\") pod \"95ca0614-d286-4cda-b2ff-30f207637882\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " Nov 04 01:33:05.751472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.751433 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\") pod \"95ca0614-d286-4cda-b2ff-30f207637882\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " Nov 04 01:33:05.751825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.751798 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\") pod \"95ca0614-d286-4cda-b2ff-30f207637882\" (UID: \"95ca0614-d286-4cda-b2ff-30f207637882\") " Nov 04 01:33:05.753839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.752731 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca" pod "95ca0614-d286-4cda-b2ff-30f207637882" (UID: "95ca0614-d286-4cda-b2ff-30f207637882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:05.761560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.761518 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca" pod "95ca0614-d286-4cda-b2ff-30f207637882" (UID: "95ca0614-d286-4cda-b2ff-30f207637882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:05.770815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.770776 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf" pod "95ca0614-d286-4cda-b2ff-30f207637882" (UID: "95ca0614-d286-4cda-b2ff-30f207637882"). InnerVolumeSpecName "kube-api-access-mcrbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:05.855723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.853650 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrbf\" (UniqueName: \"kubernetes.io/projected/95ca0614-d286-4cda-b2ff-30f207637882-kube-api-access-mcrbf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.855723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.853779 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:05.855723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:05.853802 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ca0614-d286-4cda-b2ff-30f207637882-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.418948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:06.418908 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" Nov 04 01:33:06.418948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:06.418929 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694" event={"ID":"95ca0614-d286-4cda-b2ff-30f207637882","Type":"ContainerDied","Data":"cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0"} Nov 04 01:33:06.419189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:06.419000 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf4a6b4613ad6c6506701d649163729fd3f6d8c0e286791cd2d340ed96eeaa0" Nov 04 01:33:13.832681 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:13.832635 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:33:20.276456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.276415 4486 scope.go:122] "RemoveContainer" containerID="fe75e7e379b2cad32143958f477fd7802cbc2ff76c4f1587ea1b7afcf3f319a6" Nov 04 01:33:20.287371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.287345 4486 scope.go:122] "RemoveContainer" containerID="d72f5217c0a335e23ba960c416defea168cd11df8fb16e0aef3fda084707a365" Nov 04 01:33:20.297665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.297640 4486 scope.go:122] "RemoveContainer" containerID="27e7cdd9fad17b23666ccd65d8b929bca6140efc3d90ef0fe7ced42006e7860d" Nov 04 01:33:20.310327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.310305 4486 scope.go:122] "RemoveContainer" containerID="2c402e479cc2609b40d62860f3eeda0eb3886c885b20609f2b4e1e0206628cf2" Nov 04 01:33:20.562772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.562736 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7"] Nov 04 01:33:20.568684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:20.568603 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e993d7-cnll7"] Nov 04 01:33:21.351633 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:21.351601 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e683758-b378-4a1d-a4e4-85f88992e54f" path="/var/lib/kubelet/pods/5e683758-b378-4a1d-a4e4-85f88992e54f/volumes" Nov 04 01:33:21.872005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:21.871940 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz"] Nov 04 01:33:21.919269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:21.919233 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-fc46df-v9xpz"] Nov 04 01:33:23.349594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:23.349560 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a198c8d3-175f-4d54-b31e-c401e3043879" path="/var/lib/kubelet/pods/a198c8d3-175f-4d54-b31e-c401e3043879/volumes" Nov 04 01:33:24.150813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:24.150770 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:33:24.151316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:24.151266 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" containerID="cri-o://97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1" gracePeriod=30 Nov 04 01:33:27.894880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:27.894844 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:33:27.895450 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:27.895124 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" containerID="cri-o://91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342" gracePeriod=30 Nov 04 01:33:29.431542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.431461 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:33:29.604747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.604307 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.604747 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.604670 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.605432 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.605745 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.606105 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.606274 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.606629 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.606768 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.609002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.606826 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\") pod \"dfc52389-ac0b-4510-8364-9868250cc9e6\" (UID: \"dfc52389-ac0b-4510-8364-9868250cc9e6\") " Nov 04 01:33:29.610045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.610007 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.612691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.612665 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.626826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.613404 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.626826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.613546 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:29.627157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.613640 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:29.627157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.613828 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.627157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.614255 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.627157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.614431 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "kube-api-access-b6kmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:29.627157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.626626 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.627456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.627424 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo" pod "dfc52389-ac0b-4510-8364-9868250cc9e6" (UID: "dfc52389-ac0b-4510-8364-9868250cc9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:29.639275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.639242 4486 generic.go:356] "Generic (PLEG): container finished" podID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerID="97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1" exitCode=0 Nov 04 01:33:29.639440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.639286 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" event={"ID":"dfc52389-ac0b-4510-8364-9868250cc9e6","Type":"ContainerDied","Data":"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1"} Nov 04 01:33:29.639440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.639317 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" Nov 04 01:33:29.639440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.639356 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54" event={"ID":"dfc52389-ac0b-4510-8364-9868250cc9e6","Type":"ContainerDied","Data":"0fc3c1666d324fec02c0c09150663587a344fa6d96673c9410b876bef160ad73"} Nov 04 01:33:29.639440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.639381 4486 scope.go:122] "RemoveContainer" containerID="97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1" Nov 04 01:33:29.654990 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.654949 4486 scope.go:122] "RemoveContainer" containerID="97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1" Nov 04 01:33:29.655405 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:29.655379 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1\": container with ID starting with 97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1 not found: ID does not exist" containerID="97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1" Nov 04 01:33:29.655496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.655414 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1"} err="failed to get container status \"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1\": rpc error: code = NotFound desc = could not find container \"97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1\": container with ID starting with 97c24102480ef942ca69a966abc811c645311ad958e795b071a61c50a85f85a1 not found: ID does not exist" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728429 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728461 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728470 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728478 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728487 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728498 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kmc\" (UniqueName: \"kubernetes.io/projected/dfc52389-ac0b-4510-8364-9868250cc9e6-kube-api-access-b6kmc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728508 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc52389-ac0b-4510-8364-9868250cc9e6-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.728519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.728516 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc52389-ac0b-4510-8364-9868250cc9e6-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:29.806090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.806051 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:33:29.815475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.815444 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-xlf54"] Nov 04 01:33:29.905816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.905781 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:33:29.909173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:29.906067 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" containerID="cri-o://9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3" gracePeriod=30 Nov 04 01:33:30.977595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.977558 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx"] Nov 04 01:33:30.978135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978119 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" Nov 04 01:33:30.978196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978136 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" Nov 04 01:33:30.978196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978163 4486 cpu_manager.go:414] "containerMap: removing container" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" Nov 04 01:33:30.978196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978168 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" Nov 04 01:33:30.978196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978182 4486 cpu_manager.go:414] "containerMap: removing container" podUID="95ca0614-d286-4cda-b2ff-30f207637882" containerName="revoke-keys" Nov 04 01:33:30.978196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978188 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ca0614-d286-4cda-b2ff-30f207637882" containerName="revoke-keys" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978200 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978205 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978294 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ca0614-d286-4cda-b2ff-30f207637882" containerName="revoke-keys" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978302 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e78acf8-cf6b-4ae2-9875-a2a7f3c176e0" containerName="payload-pre-processing" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978313 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" containerName="istio-proxy" Nov 04 01:33:30.978358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.978321 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ae019e-b51e-4576-9487-4ebc38c89d1a" containerName="payload-processing" Nov 04 01:33:30.981412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:30.981393 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.050475 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.050418 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.050716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.050586 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.050716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.050672 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb97f\" (UniqueName: \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.081018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.079269 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:31.081018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.079472 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:33:31.084901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.084864 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx"] Nov 04 01:33:31.152326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.152287 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.152558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.152370 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb97f\" (UniqueName: \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.152558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.152436 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.153191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.153163 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.157177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.157149 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.210700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.210662 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb97f\" (UniqueName: \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\") pod \"maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.294482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.294391 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:31.350625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.350594 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc52389-ac0b-4510-8364-9868250cc9e6" path="/var/lib/kubelet/pods/dfc52389-ac0b-4510-8364-9868250cc9e6/volumes" Nov 04 01:33:31.675068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:31.675035 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx"] Nov 04 01:33:31.675927 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:33:31.675903 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27607466_af76_4b36_990c_1cbb9a4a98a6.slice/crio-9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342 WatchSource:0}: Error finding container 9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342: Status 404 returned error can't find the container with id 9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342 Nov 04 01:33:32.666691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:32.666659 4486 generic.go:356] "Generic (PLEG): container finished" podID="27607466-af76-4b36-990c-1cbb9a4a98a6" containerID="82769f51317645d859b0afbb603a63bc4f20f5a5e83594e2631647e50d851d71" exitCode=0 Nov 04 01:33:32.667136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:32.666742 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" event={"ID":"27607466-af76-4b36-990c-1cbb9a4a98a6","Type":"ContainerDied","Data":"82769f51317645d859b0afbb603a63bc4f20f5a5e83594e2631647e50d851d71"} Nov 04 01:33:32.667136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:32.666804 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" event={"ID":"27607466-af76-4b36-990c-1cbb9a4a98a6","Type":"ContainerStarted","Data":"9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342"} Nov 04 01:33:33.193676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.193643 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:33:33.276684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.276650 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.276870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.276698 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.276870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.276746 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.276967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.276861 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.276967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.276910 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.277104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277081 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.277165 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277105 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.277207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277164 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:33.277207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277169 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.277330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277280 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.277417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277339 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\") pod \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\" (UID: \"04f016a1-5ed0-4a05-93be-fdacdaf00a3d\") " Nov 04 01:33:33.277485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277417 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.277621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277598 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.277879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277858 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.277879 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277869 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.278085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277890 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.278085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277905 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.278085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.277918 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.279349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.279323 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.279504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.279481 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "kube-api-access-mljxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.279744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.279714 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:33.279744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.279728 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy" pod "04f016a1-5ed0-4a05-93be-fdacdaf00a3d" (UID: "04f016a1-5ed0-4a05-93be-fdacdaf00a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:33.378923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.378882 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.378923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.378921 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.379223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.378937 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.379223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.378950 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.379223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.378965 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljxl\" (UniqueName: \"kubernetes.io/projected/04f016a1-5ed0-4a05-93be-fdacdaf00a3d-kube-api-access-mljxl\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.677317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.677279 4486 generic.go:356] "Generic (PLEG): container finished" podID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerID="91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342" exitCode=0 Nov 04 01:33:33.677758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.677340 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" Nov 04 01:33:33.677758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.677355 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" event={"ID":"04f016a1-5ed0-4a05-93be-fdacdaf00a3d","Type":"ContainerDied","Data":"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342"} Nov 04 01:33:33.677758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.677440 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p" event={"ID":"04f016a1-5ed0-4a05-93be-fdacdaf00a3d","Type":"ContainerDied","Data":"6caf6e654c5402a92d72b4354da4e1f162fe31c004d9b871471828e2191c2418"} Nov 04 01:33:33.677758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.677467 4486 scope.go:122] "RemoveContainer" containerID="91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342" Nov 04 01:33:33.689690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.689667 4486 scope.go:122] "RemoveContainer" containerID="91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342" Nov 04 01:33:33.690025 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:33.689971 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342\": container with ID starting with 91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342 not found: ID does not exist" containerID="91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342" Nov 04 01:33:33.690142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.690033 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342"} err="failed to get container status \"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342\": rpc error: code = NotFound desc = could not find container \"91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342\": container with ID starting with 91f1ee38d785c6128ce2eb4ad7a18ba8f5b17a2267a2bb8fdd2cfd3676107342 not found: ID does not exist" Nov 04 01:33:33.823628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.823603 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:33.840220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.840184 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:33:33.885683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.885647 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\") pod \"27607466-af76-4b36-990c-1cbb9a4a98a6\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " Nov 04 01:33:33.885890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.885696 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\") pod \"27607466-af76-4b36-990c-1cbb9a4a98a6\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " Nov 04 01:33:33.885890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.885772 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\" (UniqueName: \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\") pod \"27607466-af76-4b36-990c-1cbb9a4a98a6\" (UID: \"27607466-af76-4b36-990c-1cbb9a4a98a6\") " Nov 04 01:33:33.886120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.886090 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca" pod "27607466-af76-4b36-990c-1cbb9a4a98a6" (UID: "27607466-af76-4b36-990c-1cbb9a4a98a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:33.886261 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.886244 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.887177 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.887150 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e993d7-openshift-default-b7f7fcb7f-zdg9p"] Nov 04 01:33:33.887787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.887764 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca" pod "27607466-af76-4b36-990c-1cbb9a4a98a6" (UID: "27607466-af76-4b36-990c-1cbb9a4a98a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:33.887948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.887922 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f" pod "27607466-af76-4b36-990c-1cbb9a4a98a6" (UID: "27607466-af76-4b36-990c-1cbb9a4a98a6"). InnerVolumeSpecName "kube-api-access-zb97f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.987935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.987849 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27607466-af76-4b36-990c-1cbb9a4a98a6-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.987935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:33.987880 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb97f\" (UniqueName: \"kubernetes.io/projected/27607466-af76-4b36-990c-1cbb9a4a98a6-kube-api-access-zb97f\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.415126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:34.415089 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:34.415306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:34.415163 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:34.686944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:34.686857 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" Nov 04 01:33:34.687366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:34.686861 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx" event={"ID":"27607466-af76-4b36-990c-1cbb9a4a98a6","Type":"ContainerDied","Data":"9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342"} Nov 04 01:33:34.687366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:34.687022 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea5a34f6e7905c63026b5e732ecbb9d207a3a0b739df2e3df524c9cd263b342" Nov 04 01:33:35.185089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.185063 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:33:35.303537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303442 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.303537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303490 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.303789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303562 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.303789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303619 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.303789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303688 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.303789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303757 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.304062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303839 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.304062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303865 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.304062 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.303899 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\") pod \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\" (UID: \"60aadfea-092a-4a5a-ad5e-c11a2eff09ff\") " Nov 04 01:33:35.304736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.304602 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:35.304736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.304702 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.305251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.305228 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.305251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.305246 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.305466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.305309 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.307574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.307532 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:35.308021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.307775 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:35.308021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.308007 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.309155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.309128 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn" pod "60aadfea-092a-4a5a-ad5e-c11a2eff09ff" (UID: "60aadfea-092a-4a5a-ad5e-c11a2eff09ff"). InnerVolumeSpecName "kube-api-access-x5hhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:35.351762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.351726 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" path="/var/lib/kubelet/pods/04f016a1-5ed0-4a05-93be-fdacdaf00a3d/volumes" Nov 04 01:33:35.405554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405514 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405551 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hhn\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-kube-api-access-x5hhn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405567 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405580 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405592 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405604 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405615 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405627 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.405799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.405639 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aadfea-092a-4a5a-ad5e-c11a2eff09ff-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:35.429874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.429837 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:33:35.434393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.434363 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:33:35.699527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.699479 4486 generic.go:356] "Generic (PLEG): container finished" podID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerID="9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3" exitCode=0 Nov 04 01:33:35.699527 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.699528 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerDied","Data":"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3"} Nov 04 01:33:35.700164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.699544 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" Nov 04 01:33:35.700164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.699575 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87" event={"ID":"60aadfea-092a-4a5a-ad5e-c11a2eff09ff","Type":"ContainerDied","Data":"5b2bfcd6a4ad7e1e40dd3320bfc8cadee20dbb36bc4ce8ec1b02e3f3f9c7322e"} Nov 04 01:33:35.700164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.699602 4486 scope.go:122] "RemoveContainer" containerID="9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3" Nov 04 01:33:35.710419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.710396 4486 scope.go:122] "RemoveContainer" containerID="020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a" Nov 04 01:33:35.720314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.720281 4486 scope.go:122] "RemoveContainer" containerID="9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3" Nov 04 01:33:35.720610 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:35.720587 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3\": container with ID starting with 9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3 not found: ID does not exist" containerID="9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3" Nov 04 01:33:35.720700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.720623 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3"} err="failed to get container status \"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3\": rpc error: code = NotFound desc = could not find container \"9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3\": container with ID starting with 9cda8b28f7645e3792f9a85bf603f8d9035b474e1f9ee957fde44c4537a5edb3 not found: ID does not exist" Nov 04 01:33:35.720700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.720644 4486 scope.go:122] "RemoveContainer" containerID="020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a" Nov 04 01:33:35.720933 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:35.720916 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a\": container with ID starting with 020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a not found: ID does not exist" containerID="020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a" Nov 04 01:33:35.720975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.720938 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a"} err="failed to get container status \"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a\": rpc error: code = NotFound desc = could not find container \"020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a\": container with ID starting with 020b6ab996c516fe6a629cd86f9e0e28d003c25551ae444fadb81527a861888a not found: ID does not exist" Nov 04 01:33:35.868794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.868760 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:33:35.872375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:35.872348 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-fc46df-openshift-default-5cf4464fd5-7qp87"] Nov 04 01:33:37.349001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:37.348942 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" path="/var/lib/kubelet/pods/60aadfea-092a-4a5a-ad5e-c11a2eff09ff/volumes" Nov 04 01:33:39.359256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:39.359217 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694"] Nov 04 01:33:39.406638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:39.406602 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-98f846-h7694"] Nov 04 01:33:41.356385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:41.356341 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ca0614-d286-4cda-b2ff-30f207637882" path="/var/lib/kubelet/pods/95ca0614-d286-4cda-b2ff-30f207637882/volumes" Nov 04 01:33:43.834788 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:43.834742 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:33:46.347972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:46.347936 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:33:46.348596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:46.348186 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" containerID="cri-o://441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f" gracePeriod=30 Nov 04 01:33:47.252493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.252447 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq"] Nov 04 01:33:47.253340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253317 4486 cpu_manager.go:414] "containerMap: removing container" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" Nov 04 01:33:47.253402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253343 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" Nov 04 01:33:47.253402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253359 4486 cpu_manager.go:414] "containerMap: removing container" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.253402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253367 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.253402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253378 4486 cpu_manager.go:414] "containerMap: removing container" podUID="27607466-af76-4b36-990c-1cbb9a4a98a6" containerName="revoke-keys" Nov 04 01:33:47.253402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253386 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27607466-af76-4b36-990c-1cbb9a4a98a6" containerName="revoke-keys" Nov 04 01:33:47.253568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253405 4486 cpu_manager.go:414] "containerMap: removing container" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.253568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253412 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.253568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253554 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f016a1-5ed0-4a05-93be-fdacdaf00a3d" containerName="istio-proxy" Nov 04 01:33:47.253568 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253565 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.253696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253576 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="27607466-af76-4b36-990c-1cbb9a4a98a6" containerName="revoke-keys" Nov 04 01:33:47.253696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.253588 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aadfea-092a-4a5a-ad5e-c11a2eff09ff" containerName="istio-proxy" Nov 04 01:33:47.258859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.258838 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.302694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.302652 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.302887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.302868 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.302939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.302925 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtgs\" (UniqueName: \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.312693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.312658 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq"] Nov 04 01:33:47.353448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.353416 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:33:47.353448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.353431 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:47.404213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.404174 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.404440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.404303 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.404440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.404346 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtgs\" (UniqueName: \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.405093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.405067 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.406708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.406687 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.459610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.459575 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtgs\" (UniqueName: \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.575036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.574963 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:47.954241 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:47.954204 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq"] Nov 04 01:33:47.955212 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:33:47.955185 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e44385f_250c_4342_85a1_bb4c0fe506ad.slice/crio-d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211 WatchSource:0}: Error finding container d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211: Status 404 returned error can't find the container with id d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211 Nov 04 01:33:48.507069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:48.506966 4486 patch_prober.go:29] interesting pod/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:48.507069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:48.507039 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:48.819682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:48.819641 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" event={"ID":"5e44385f-250c-4342-85a1-bb4c0fe506ad","Type":"ContainerStarted","Data":"144b97d0a66af5adab7026f76f6d3bf16871d3ae87083262654a531ba35b6f7b"} Nov 04 01:33:48.820105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:48.819701 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" event={"ID":"5e44385f-250c-4342-85a1-bb4c0fe506ad","Type":"ContainerStarted","Data":"d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211"} Nov 04 01:33:48.938064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:48.937973 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" podStartSLOduration=1.9379570940000002 podStartE2EDuration="1.937957094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:48.934652337 +0000 UTC m=+2197.693314498" watchObservedRunningTime="2026-11-04 01:33:48.937957094 +0000 UTC m=+2197.696619657" Nov 04 01:33:49.154146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.154066 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq"] Nov 04 01:33:49.158358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.158333 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.215374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.215333 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq"] Nov 04 01:33:49.225954 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.225919 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nd5\" (UniqueName: \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.226161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.226072 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.226161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.226136 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.327064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.327024 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.327303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.327095 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.327303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.327203 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nd5\" (UniqueName: \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.327770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.327735 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.329559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.329527 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.384406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.384364 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nd5\" (UniqueName: \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\") pod \"maas-api-revoke-sub-524a85d0-nf8zq\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.472809 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.472717 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:49.829530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.829488 4486 generic.go:356] "Generic (PLEG): container finished" podID="5e44385f-250c-4342-85a1-bb4c0fe506ad" containerID="144b97d0a66af5adab7026f76f6d3bf16871d3ae87083262654a531ba35b6f7b" exitCode=0 Nov 04 01:33:49.829906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.829569 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" event={"ID":"5e44385f-250c-4342-85a1-bb4c0fe506ad","Type":"ContainerDied","Data":"144b97d0a66af5adab7026f76f6d3bf16871d3ae87083262654a531ba35b6f7b"} Nov 04 01:33:49.859790 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:33:49.859745 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbde716_c40d_4ef9_8d20_32f72cd651bc.slice/crio-5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d WatchSource:0}: Error finding container 5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d: Status 404 returned error can't find the container with id 5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d Nov 04 01:33:49.860440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:49.860413 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq"] Nov 04 01:33:50.059477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.059441 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4"] Nov 04 01:33:50.073855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.073811 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2"] Nov 04 01:33:50.078280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.078231 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5af534c-nzbp4"] Nov 04 01:33:50.083409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.083332 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1b61114-mjfs2"] Nov 04 01:33:50.451908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.451811 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" containerID="cri-o://260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71" gracePeriod=45 Nov 04 01:33:50.454919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.454880 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" containerID="cri-o://a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c" gracePeriod=45 Nov 04 01:33:50.585405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.585366 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/ready\": dial tcp 10.133.0.109:8000: connect: connection refused" Nov 04 01:33:50.755644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.755617 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:33:50.759441 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.759415 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:33:50.841729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.841691 4486 generic.go:356] "Generic (PLEG): container finished" podID="3fbde716-c40d-4ef9-8d20-32f72cd651bc" containerID="e5322b0121ebaeb84a8f85856da8a12de3a2053b14f2031190b78ff7e27c5153" exitCode=0 Nov 04 01:33:50.842307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.841779 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" event={"ID":"3fbde716-c40d-4ef9-8d20-32f72cd651bc","Type":"ContainerDied","Data":"e5322b0121ebaeb84a8f85856da8a12de3a2053b14f2031190b78ff7e27c5153"} Nov 04 01:33:50.842307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.841851 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" event={"ID":"3fbde716-c40d-4ef9-8d20-32f72cd651bc","Type":"ContainerStarted","Data":"5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d"} Nov 04 01:33:50.843833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.843799 4486 generic.go:356] "Generic (PLEG): container finished" podID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerID="260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71" exitCode=0 Nov 04 01:33:50.843967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.843848 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" Nov 04 01:33:50.843967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.843889 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" event={"ID":"3c96b0f2-11e4-444e-b064-9ef968de5399","Type":"ContainerDied","Data":"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71"} Nov 04 01:33:50.843967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.843956 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z" event={"ID":"3c96b0f2-11e4-444e-b064-9ef968de5399","Type":"ContainerDied","Data":"fe2abcdae7c306401913654add87da2e6116fd8b5216d33dfd4f92eb55046ce9"} Nov 04 01:33:50.844198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.844005 4486 scope.go:122] "RemoveContainer" containerID="260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71" Nov 04 01:33:50.846878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.846770 4486 generic.go:356] "Generic (PLEG): container finished" podID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerID="a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c" exitCode=0 Nov 04 01:33:50.846878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.846801 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" event={"ID":"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443","Type":"ContainerDied","Data":"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c"} Nov 04 01:33:50.846878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.846858 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" event={"ID":"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443","Type":"ContainerDied","Data":"416857b17ae7bfd5737c5ccb150fccdd12ff21ee8c4fd9fa866550697a57a03c"} Nov 04 01:33:50.847229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847015 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.847229 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847084 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.847346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847233 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.847346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847271 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\" (UniqueName: \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.847346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847307 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.847346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847342 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\" (UniqueName: \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847363 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847382 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847478 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847505 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\") pod \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\" (UID: \"d1fb6f25-ef8f-4d1a-ba0b-5efedab64443\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847535 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847572 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.847653 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\") pod \"3c96b0f2-11e4-444e-b064-9ef968de5399\" (UID: \"3c96b0f2-11e4-444e-b064-9ef968de5399\") " Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.848190 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.848384 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.848500 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.848684 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.848785 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.851327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.849191 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.852419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.852384 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb" Nov 04 01:33:50.852845 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.852809 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.853266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.853239 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "kube-api-access-889fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:50.858947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.858906 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:50.858947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.858938 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "kube-api-access-w76h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:50.860573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.859610 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs" pod "3c96b0f2-11e4-444e-b064-9ef968de5399" (UID: "3c96b0f2-11e4-444e-b064-9ef968de5399"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:50.860873 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.860848 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs" pod "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" (UID: "d1fb6f25-ef8f-4d1a-ba0b-5efedab64443"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:50.881470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.881437 4486 scope.go:122] "RemoveContainer" containerID="260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71" Nov 04 01:33:50.881940 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:50.881905 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71\": container with ID starting with 260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71 not found: ID does not exist" containerID="260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71" Nov 04 01:33:50.882102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.881947 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71"} err="failed to get container status \"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71\": rpc error: code = NotFound desc = could not find container \"260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71\": container with ID starting with 260ee3728ed00bdf84102dfe1f55265b003520ba997a3d7204344348223d9c71 not found: ID does not exist" Nov 04 01:33:50.882102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.881967 4486 scope.go:122] "RemoveContainer" containerID="a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c" Nov 04 01:33:50.901303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.901275 4486 scope.go:122] "RemoveContainer" containerID="a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c" Nov 04 01:33:50.901744 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:50.901704 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c\": container with ID starting with a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c not found: ID does not exist" containerID="a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c" Nov 04 01:33:50.906100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.901744 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c"} err="failed to get container status \"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c\": rpc error: code = NotFound desc = could not find container \"a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c\": container with ID starting with a2ffc622a5a2b72dafab0968d93bbb46aa50ed5c141e66ad1fa00be635cafd4c not found: ID does not exist" Nov 04 01:33:50.949426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949375 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949419 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949435 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949452 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949467 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949482 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c96b0f2-11e4-444e-b064-9ef968de5399-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949495 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949507 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889fr\" (UniqueName: \"kubernetes.io/projected/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-kube-api-access-889fr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949518 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c96b0f2-11e4-444e-b064-9ef968de5399-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949527 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w76h7\" (UniqueName: \"kubernetes.io/projected/3c96b0f2-11e4-444e-b064-9ef968de5399-kube-api-access-w76h7\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:50.949674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:50.949536 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.019101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.019072 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:51.050812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.050771 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\" (UniqueName: \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\") pod \"5e44385f-250c-4342-85a1-bb4c0fe506ad\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " Nov 04 01:33:51.051058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.050941 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\") pod \"5e44385f-250c-4342-85a1-bb4c0fe506ad\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " Nov 04 01:33:51.051058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.050967 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\") pod \"5e44385f-250c-4342-85a1-bb4c0fe506ad\" (UID: \"5e44385f-250c-4342-85a1-bb4c0fe506ad\") " Nov 04 01:33:51.051384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.051356 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca" pod "5e44385f-250c-4342-85a1-bb4c0fe506ad" (UID: "5e44385f-250c-4342-85a1-bb4c0fe506ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:51.053141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.053118 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca" pod "5e44385f-250c-4342-85a1-bb4c0fe506ad" (UID: "5e44385f-250c-4342-85a1-bb4c0fe506ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:51.053218 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.053170 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs" pod "5e44385f-250c-4342-85a1-bb4c0fe506ad" (UID: "5e44385f-250c-4342-85a1-bb4c0fe506ad"). InnerVolumeSpecName "kube-api-access-qwtgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:51.152312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.152278 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtgs\" (UniqueName: \"kubernetes.io/projected/5e44385f-250c-4342-85a1-bb4c0fe506ad-kube-api-access-qwtgs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.152312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.152307 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.152312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.152318 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e44385f-250c-4342-85a1-bb4c0fe506ad-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.277792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.277701 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:33:51.285602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.285569 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-w5-e210ef-kserve-596bdbf6cc-w7wpb"] Nov 04 01:33:51.349144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.349099 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a076ce-769e-47fa-a775-a5bff8622f77" path="/var/lib/kubelet/pods/93a076ce-769e-47fa-a775-a5bff8622f77/volumes" Nov 04 01:33:51.349573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.349558 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c616ad-ef19-4989-9d7f-55e2dff35fd0" path="/var/lib/kubelet/pods/a3c616ad-ef19-4989-9d7f-55e2dff35fd0/volumes" Nov 04 01:33:51.349948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.349935 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" path="/var/lib/kubelet/pods/d1fb6f25-ef8f-4d1a-ba0b-5efedab64443/volumes" Nov 04 01:33:51.392401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.392362 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:33:51.399466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.399430 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e210ef/distinct-2-w5-e210ef-kserve-664795465d-5856z"] Nov 04 01:33:51.627552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.627524 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:33:51.758389 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758356 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758407 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758470 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758501 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758543 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758616 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758664 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758701 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.758803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758731 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\") pod \"b3af9229-7b9f-4553-8896-b66f6a75b146\" (UID: \"b3af9229-7b9f-4553-8896-b66f6a75b146\") " Nov 04 01:33:51.759018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758696 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:51.759103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.758951 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:51.759267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.759248 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.759338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.759273 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.759511 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.759493 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:51.759603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.759554 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:51.759731 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.759712 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:51.761280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.761256 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:51.761442 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.761418 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:51.761564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.761541 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:51.762252 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.762229 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w" pod "b3af9229-7b9f-4553-8896-b66f6a75b146" (UID: "b3af9229-7b9f-4553-8896-b66f6a75b146"). InnerVolumeSpecName "kube-api-access-24l9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:51.858129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.858098 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" Nov 04 01:33:51.858550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.858137 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq" event={"ID":"5e44385f-250c-4342-85a1-bb4c0fe506ad","Type":"ContainerDied","Data":"d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211"} Nov 04 01:33:51.858550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.858193 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d98b7d0c18f353e074335b13b60e30542bb237583319f39174f6b757c2aa2211" Nov 04 01:33:51.859953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.859924 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24l9w\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-kube-api-access-24l9w\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.859957 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.859971 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860004 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3af9229-7b9f-4553-8896-b66f6a75b146-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860019 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860032 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3af9229-7b9f-4553-8896-b66f6a75b146-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860048 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3af9229-7b9f-4553-8896-b66f6a75b146-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:51.860512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860201 4486 generic.go:356] "Generic (PLEG): container finished" podID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerID="441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f" exitCode=0 Nov 04 01:33:51.860512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860249 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" Nov 04 01:33:51.860512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860275 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerDied","Data":"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f"} Nov 04 01:33:51.860512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860331 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw" event={"ID":"b3af9229-7b9f-4553-8896-b66f6a75b146","Type":"ContainerDied","Data":"0ed189b748605a69e15385f9a79ed443064adbc69ad26b9a319c1fae80f7e8c1"} Nov 04 01:33:51.860512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.860364 4486 scope.go:122] "RemoveContainer" containerID="441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f" Nov 04 01:33:51.872493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.872470 4486 scope.go:122] "RemoveContainer" containerID="e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb" Nov 04 01:33:51.883220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.883195 4486 scope.go:122] "RemoveContainer" containerID="441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f" Nov 04 01:33:51.883524 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:51.883498 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f\": container with ID starting with 441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f not found: ID does not exist" containerID="441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f" Nov 04 01:33:51.883653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.883530 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f"} err="failed to get container status \"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f\": rpc error: code = NotFound desc = could not find container \"441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f\": container with ID starting with 441935c6c3b31c151954660bd2ea0b2e62056c0721a3f935a5a502ed261bf46f not found: ID does not exist" Nov 04 01:33:51.883653 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.883548 4486 scope.go:122] "RemoveContainer" containerID="e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb" Nov 04 01:33:51.883856 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:33:51.883831 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb\": container with ID starting with e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb not found: ID does not exist" containerID="e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb" Nov 04 01:33:51.883920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.883860 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb"} err="failed to get container status \"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb\": rpc error: code = NotFound desc = could not find container \"e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb\": container with ID starting with e10b134b5a161c620412c5a5bedc1473319ce7de11dc1755ecd46e418e51e5eb not found: ID does not exist" Nov 04 01:33:51.999563 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:51.999536 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:52.022248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.022215 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:33:52.029296 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.029266 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-98f846-openshift-default-5958cf69c7-fvhbw"] Nov 04 01:33:52.062240 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.062201 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\" (UniqueName: \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\") pod \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " Nov 04 01:33:52.062427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.062335 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\") pod \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " Nov 04 01:33:52.062427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.062406 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\") pod \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\" (UID: \"3fbde716-c40d-4ef9-8d20-32f72cd651bc\") " Nov 04 01:33:52.062806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.062777 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca" pod "3fbde716-c40d-4ef9-8d20-32f72cd651bc" (UID: "3fbde716-c40d-4ef9-8d20-32f72cd651bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:52.064415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.064388 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca" pod "3fbde716-c40d-4ef9-8d20-32f72cd651bc" (UID: "3fbde716-c40d-4ef9-8d20-32f72cd651bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:52.064415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.064390 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5" pod "3fbde716-c40d-4ef9-8d20-32f72cd651bc" (UID: "3fbde716-c40d-4ef9-8d20-32f72cd651bc"). InnerVolumeSpecName "kube-api-access-v4nd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:52.163802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.163711 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nd5\" (UniqueName: \"kubernetes.io/projected/3fbde716-c40d-4ef9-8d20-32f72cd651bc-kube-api-access-v4nd5\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:52.163802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.163748 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:52.163802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.163758 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbde716-c40d-4ef9-8d20-32f72cd651bc-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:52.872449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.872417 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" Nov 04 01:33:52.872894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.872447 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq" event={"ID":"3fbde716-c40d-4ef9-8d20-32f72cd651bc","Type":"ContainerDied","Data":"5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d"} Nov 04 01:33:52.872894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:52.872500 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5503ddff5357ca541e36ab6291d764e57efabb46aa4a5ae53011d007070c190d" Nov 04 01:33:53.350087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:53.350046 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" path="/var/lib/kubelet/pods/3c96b0f2-11e4-444e-b064-9ef968de5399/volumes" Nov 04 01:33:53.350555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:53.350541 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" path="/var/lib/kubelet/pods/b3af9229-7b9f-4553-8896-b66f6a75b146/volumes" Nov 04 01:33:58.898046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:58.898004 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:33:59.003166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:33:59.003124 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:34:03.898754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:03.898710 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" podUID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" containerName="payload-processing" containerID="cri-o://1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda" gracePeriod=25 Nov 04 01:34:04.003850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.003798 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" podUID="10e227e1-8243-46fb-aa11-b4170b24c321" containerName="payload-pre-processing" containerID="cri-o://6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4" gracePeriod=25 Nov 04 01:34:04.163189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.163120 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:34:04.278157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.278129 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:34:04.326880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.326827 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\" (UniqueName: \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\") pod \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " Nov 04 01:34:04.327169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.326941 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\") pod \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\" (UID: \"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6\") " Nov 04 01:34:04.327719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.327687 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume" pod "f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" (UID: "f583c6e3-cfa3-464c-aa8c-29ad6e111ba6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:04.329489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.329462 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc" pod "f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" (UID: "f583c6e3-cfa3-464c-aa8c-29ad6e111ba6"). InnerVolumeSpecName "kube-api-access-d2hnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.428607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.428502 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\") pod \"10e227e1-8243-46fb-aa11-b4170b24c321\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " Nov 04 01:34:04.428796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.428673 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\" (UniqueName: \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\") pod \"10e227e1-8243-46fb-aa11-b4170b24c321\" (UID: \"10e227e1-8243-46fb-aa11-b4170b24c321\") " Nov 04 01:34:04.429128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.429099 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2hnc\" (UniqueName: \"kubernetes.io/projected/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-kube-api-access-d2hnc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.429128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.429122 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.429333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.429266 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume" pod "10e227e1-8243-46fb-aa11-b4170b24c321" (UID: "10e227e1-8243-46fb-aa11-b4170b24c321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:04.431662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.431613 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2" pod "10e227e1-8243-46fb-aa11-b4170b24c321" (UID: "10e227e1-8243-46fb-aa11-b4170b24c321"). InnerVolumeSpecName "kube-api-access-45tm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.530752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.530716 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10e227e1-8243-46fb-aa11-b4170b24c321-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.530752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.530749 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45tm2\" (UniqueName: \"kubernetes.io/projected/10e227e1-8243-46fb-aa11-b4170b24c321-kube-api-access-45tm2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.975757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.975715 4486 generic.go:356] "Generic (PLEG): container finished" podID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" containerID="1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda" exitCode=0 Nov 04 01:34:04.976302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.975761 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" Nov 04 01:34:04.976302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.975801 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" event={"ID":"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6","Type":"ContainerDied","Data":"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda"} Nov 04 01:34:04.976302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.975865 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd" event={"ID":"f583c6e3-cfa3-464c-aa8c-29ad6e111ba6","Type":"ContainerDied","Data":"be3e676da219a04d00d7279d53478ceb7a49be443b2f08131bbb8b5ba6d10f1f"} Nov 04 01:34:04.976302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.975897 4486 scope.go:122] "RemoveContainer" containerID="1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda" Nov 04 01:34:04.977460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.977436 4486 generic.go:356] "Generic (PLEG): container finished" podID="10e227e1-8243-46fb-aa11-b4170b24c321" containerID="6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4" exitCode=0 Nov 04 01:34:04.977552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.977486 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" Nov 04 01:34:04.977552 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.977520 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" event={"ID":"10e227e1-8243-46fb-aa11-b4170b24c321","Type":"ContainerDied","Data":"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4"} Nov 04 01:34:04.977651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.977577 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx" event={"ID":"10e227e1-8243-46fb-aa11-b4170b24c321","Type":"ContainerDied","Data":"8b71383479b9c3a76dc627ac4b934a67a88332ffae32d728e1d9022738a6e18e"} Nov 04 01:34:04.988175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.988152 4486 scope.go:122] "RemoveContainer" containerID="1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda" Nov 04 01:34:04.988549 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:04.988511 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda\": container with ID starting with 1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda not found: ID does not exist" containerID="1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda" Nov 04 01:34:04.988642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.988557 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda"} err="failed to get container status \"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda\": rpc error: code = NotFound desc = could not find container \"1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda\": container with ID starting with 1e83ee06590dc32b1924e066f1b71fa967e41d0967e9db9902fb8c72b6471eda not found: ID does not exist" Nov 04 01:34:04.988642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.988576 4486 scope.go:122] "RemoveContainer" containerID="6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4" Nov 04 01:34:04.998638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.998619 4486 scope.go:122] "RemoveContainer" containerID="6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4" Nov 04 01:34:04.998956 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:04.998937 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4\": container with ID starting with 6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4 not found: ID does not exist" containerID="6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4" Nov 04 01:34:04.999025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:04.998969 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4"} err="failed to get container status \"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4\": rpc error: code = NotFound desc = could not find container \"6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4\": container with ID starting with 6f8ccf81634fedd13f1d6b6e252e9c4e2ee832d9a07ee96f4f3af0eae71869a4 not found: ID does not exist" Nov 04 01:34:05.136468 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.136432 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:34:05.183715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.183630 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-843be0-5f8fbf447-h26rx"] Nov 04 01:34:05.243366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.243329 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:34:05.246670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.246640 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-843be0-5bc98ccbc8-2nvrd"] Nov 04 01:34:05.349084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.349052 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e227e1-8243-46fb-aa11-b4170b24c321" path="/var/lib/kubelet/pods/10e227e1-8243-46fb-aa11-b4170b24c321/volumes" Nov 04 01:34:05.349482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:05.349468 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" path="/var/lib/kubelet/pods/f583c6e3-cfa3-464c-aa8c-29ad6e111ba6/volumes" Nov 04 01:34:07.367506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:07.367470 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx"] Nov 04 01:34:07.414553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:07.414515 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e210ef-rz8cx"] Nov 04 01:34:09.349325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:09.349285 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27607466-af76-4b36-990c-1cbb9a4a98a6" path="/var/lib/kubelet/pods/27607466-af76-4b36-990c-1cbb9a4a98a6/volumes" Nov 04 01:34:13.850814 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:13.850753 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:34:14.020325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:14.020289 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq"] Nov 04 01:34:14.067546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:14.067511 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-843be0-h9xrq"] Nov 04 01:34:15.349456 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:15.349417 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e44385f-250c-4342-85a1-bb4c0fe506ad" path="/var/lib/kubelet/pods/5e44385f-250c-4342-85a1-bb4c0fe506ad/volumes" Nov 04 01:34:16.010739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:16.010699 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:34:16.011036 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:16.010975 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" containerID="cri-o://78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff" gracePeriod=30 Nov 04 01:34:20.418896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:20.418865 4486 scope.go:122] "RemoveContainer" containerID="4fae13aa8890db2144d3c7561879a6ace380248f0d150523f1e82500fd21a732" Nov 04 01:34:20.428862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:20.428832 4486 scope.go:122] "RemoveContainer" containerID="767699ea2474c1f8f321f75baded690b7bf88a4638f4f806ba789273e3ae914b" Nov 04 01:34:20.441019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:20.440974 4486 scope.go:122] "RemoveContainer" containerID="ea900b1f1c048738430bdfd7fd4ffa2d9e548e06020e3eab09182e82eeafa530" Nov 04 01:34:20.450305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:20.450281 4486 scope.go:122] "RemoveContainer" containerID="613abca881219e7b58f73703e57e634b3031ef0aa167beed008086046a18193a" Nov 04 01:34:21.286108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.286032 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:34:21.418565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418526 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.418772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418579 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.418772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418640 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.418772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418686 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.418772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418732 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418780 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418872 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418903 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418935 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.418964 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\") pod \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\" (UID: \"0b0f9407-291d-4c95-a83d-f4b5f88fce9a\") " Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419099 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419138 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:21.419334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419262 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419382 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419555 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419574 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419589 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419605 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.419752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.419621 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.421140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.421108 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "kube-api-access-rpvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:21.421491 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.421466 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:21.421830 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.421800 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:21.421937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.421874 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token" pod "0b0f9407-291d-4c95-a83d-f4b5f88fce9a" (UID: "0b0f9407-291d-4c95-a83d-f4b5f88fce9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:21.520721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.520684 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.520721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.520719 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvxv\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-kube-api-access-rpvxv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.520942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.520731 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:21.520942 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:21.520741 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f9407-291d-4c95-a83d-f4b5f88fce9a-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:22.112654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.112614 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:34:22.112877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.112849 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" containerID="cri-o://a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09" gracePeriod=30 Nov 04 01:34:22.127106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.127074 4486 generic.go:356] "Generic (PLEG): container finished" podID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerID="78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff" exitCode=0 Nov 04 01:34:22.127281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.127139 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerDied","Data":"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff"} Nov 04 01:34:22.127281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.127190 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" Nov 04 01:34:22.127281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.127206 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2" event={"ID":"0b0f9407-291d-4c95-a83d-f4b5f88fce9a","Type":"ContainerDied","Data":"7f1bfc6da6d36c0e6bbf82c49bfe1a7659c6ad89a686b09d197b3dc299af5a75"} Nov 04 01:34:22.127281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.127229 4486 scope.go:122] "RemoveContainer" containerID="78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff" Nov 04 01:34:22.139228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.139202 4486 scope.go:122] "RemoveContainer" containerID="d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8" Nov 04 01:34:22.175417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.175385 4486 scope.go:122] "RemoveContainer" containerID="78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff" Nov 04 01:34:22.175768 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:22.175748 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff\": container with ID starting with 78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff not found: ID does not exist" containerID="78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff" Nov 04 01:34:22.175852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.175778 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff"} err="failed to get container status \"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff\": rpc error: code = NotFound desc = could not find container \"78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff\": container with ID starting with 78da1c18a1cc6fb1d1062bf6275adf86b3924917a3abd9d7dbbf7791417fbaff not found: ID does not exist" Nov 04 01:34:22.175852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.175800 4486 scope.go:122] "RemoveContainer" containerID="d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8" Nov 04 01:34:22.176133 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:22.176113 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8\": container with ID starting with d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8 not found: ID does not exist" containerID="d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8" Nov 04 01:34:22.176192 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.176174 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8"} err="failed to get container status \"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8\": rpc error: code = NotFound desc = could not find container \"d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8\": container with ID starting with d6a208837f82827008cbd5561e47389efe6da27daf2eeee86cf113a3ce8755a8 not found: ID does not exist" Nov 04 01:34:22.344108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.343746 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:34:22.347847 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:22.347818 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e210ef-openshift-default-bc95699cc-mgng2"] Nov 04 01:34:23.349393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:23.349350 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" path="/var/lib/kubelet/pods/0b0f9407-291d-4c95-a83d-f4b5f88fce9a/volumes" Nov 04 01:34:25.360350 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:25.360310 4486 patch_prober.go:29] interesting pod/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:25.360770 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:25.360384 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:27.393927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.393853 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:34:27.483741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483707 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483754 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483793 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483814 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483869 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483890 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.483948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.483930 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484011 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484051 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\") pod \"12a87968-6e6d-4188-a8d5-1d03cc748fba\" (UID: \"12a87968-6e6d-4188-a8d5-1d03cc748fba\") " Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484061 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484147 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484177 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.484314 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484197 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.484640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484484 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:27.484705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484648 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.484705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484671 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.484705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484686 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.484705 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484698 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.485023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.484712 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a87968-6e6d-4188-a8d5-1d03cc748fba-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.486208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.486185 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:27.486311 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.486297 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "kube-api-access-dbchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:27.486579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.486553 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:27.486683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.486642 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy" pod "12a87968-6e6d-4188-a8d5-1d03cc748fba" (UID: "12a87968-6e6d-4188-a8d5-1d03cc748fba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:27.586233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.586198 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.586233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.586226 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.586233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.586236 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbchs\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-kube-api-access-dbchs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.586486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.586245 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a87968-6e6d-4188-a8d5-1d03cc748fba-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:27.618785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:27.618704 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:34:28.175649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.175612 4486 generic.go:356] "Generic (PLEG): container finished" podID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerID="a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09" exitCode=0 Nov 04 01:34:28.175857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.175700 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" Nov 04 01:34:28.175857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.175692 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerDied","Data":"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09"} Nov 04 01:34:28.176024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.175858 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs" event={"ID":"12a87968-6e6d-4188-a8d5-1d03cc748fba","Type":"ContainerDied","Data":"9e8ba61ace4d4daf94ce991742280a36eb73aec04ce4e8f3483c6f48bbe18b78"} Nov 04 01:34:28.176024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.175887 4486 scope.go:122] "RemoveContainer" containerID="a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09" Nov 04 01:34:28.189483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.189460 4486 scope.go:122] "RemoveContainer" containerID="2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f" Nov 04 01:34:28.201077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.201053 4486 scope.go:122] "RemoveContainer" containerID="a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09" Nov 04 01:34:28.201404 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:28.201385 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09\": container with ID starting with a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09 not found: ID does not exist" containerID="a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09" Nov 04 01:34:28.201458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.201418 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09"} err="failed to get container status \"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09\": rpc error: code = NotFound desc = could not find container \"a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09\": container with ID starting with a38a3c14872146d4e8228f550828d0efeb5b96ef11d70542b28a402977d56e09 not found: ID does not exist" Nov 04 01:34:28.201458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.201436 4486 scope.go:122] "RemoveContainer" containerID="2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f" Nov 04 01:34:28.201704 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:28.201687 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f\": container with ID starting with 2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f not found: ID does not exist" containerID="2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f" Nov 04 01:34:28.201752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.201710 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f"} err="failed to get container status \"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f\": rpc error: code = NotFound desc = could not find container \"2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f\": container with ID starting with 2d49d974ca5ad28b8080331b1f4330bbd12f755668b0dfdc2e12a18e5527ce2f not found: ID does not exist" Nov 04 01:34:28.384531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.384479 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:34:28.430888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:28.430796 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-843be0-openshift-default-7ffdc55b9-blmhs"] Nov 04 01:34:29.349024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:29.348961 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" path="/var/lib/kubelet/pods/12a87968-6e6d-4188-a8d5-1d03cc748fba/volumes" Nov 04 01:34:34.837213 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:34.837163 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:34:34.950197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:34.950167 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:34:39.169996 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.169949 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f"] Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170511 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170525 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170547 4486 cpu_manager.go:414] "containerMap: removing container" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170553 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170560 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170565 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170576 4486 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170581 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170587 4486 cpu_manager.go:414] "containerMap: removing container" podUID="5e44385f-250c-4342-85a1-bb4c0fe506ad" containerName="revoke-keys" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170593 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e44385f-250c-4342-85a1-bb4c0fe506ad" containerName="revoke-keys" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170609 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3fbde716-c40d-4ef9-8d20-32f72cd651bc" containerName="revoke-keys" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170616 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbde716-c40d-4ef9-8d20-32f72cd651bc" containerName="revoke-keys" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170627 4486 cpu_manager.go:414] "containerMap: removing container" podUID="10e227e1-8243-46fb-aa11-b4170b24c321" containerName="payload-pre-processing" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170632 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e227e1-8243-46fb-aa11-b4170b24c321" containerName="payload-pre-processing" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170641 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" Nov 04 01:34:39.170638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170646 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170655 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170659 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170668 4486 cpu_manager.go:414] "containerMap: removing container" podUID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" containerName="payload-processing" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170672 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" containerName="payload-processing" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170750 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="f583c6e3-cfa3-464c-aa8c-29ad6e111ba6" containerName="payload-processing" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170760 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170767 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170777 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170786 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170791 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170797 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e44385f-250c-4342-85a1-bb4c0fe506ad" containerName="revoke-keys" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170802 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f9407-291d-4c95-a83d-f4b5f88fce9a" containerName="istio-proxy" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170809 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbde716-c40d-4ef9-8d20-32f72cd651bc" containerName="revoke-keys" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170814 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e227e1-8243-46fb-aa11-b4170b24c321" containerName="payload-pre-processing" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170820 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fb6f25-ef8f-4d1a-ba0b-5efedab64443" containerName="main" Nov 04 01:34:39.171561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.170825 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c96b0f2-11e4-444e-b064-9ef968de5399" containerName="main" Nov 04 01:34:39.173958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.173939 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.238256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.238223 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:34:39.283558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.283526 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:34:39.286741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.286713 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f"] Nov 04 01:34:39.318521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.318483 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtj2m\" (UniqueName: \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.318714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.318682 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.318785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.318745 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.420521 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.420417 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtj2m\" (UniqueName: \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.420698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.420541 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.420698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.420596 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.421333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.421306 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.423115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.423093 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.477430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.477400 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtj2m\" (UniqueName: \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\") pod \"maas-api-revoke-sub-e6be64d0-m7j4f\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.487232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.487203 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:39.837531 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.837487 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" containerID="cri-o://4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d" gracePeriod=25 Nov 04 01:34:39.860550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.860360 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f"] Nov 04 01:34:39.861842 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:34:39.861804 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ad2415_abb7_4e2e_bdb2_fcb815fd39b0.slice/crio-9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c WatchSource:0}: Error finding container 9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c: Status 404 returned error can't find the container with id 9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c Nov 04 01:34:39.951256 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:39.951192 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" containerID="cri-o://a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c" gracePeriod=25 Nov 04 01:34:40.265275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:40.265234 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" event={"ID":"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0","Type":"ContainerStarted","Data":"2190e660987deb48f6abdaee215aeee8e4842976fdbad6a99810ba5fb11b5f46"} Nov 04 01:34:40.265693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:40.265293 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" event={"ID":"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0","Type":"ContainerStarted","Data":"9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c"} Nov 04 01:34:40.378699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:40.378638 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" podStartSLOduration=1.378609438 podStartE2EDuration="1.378609438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:40.377449337 +0000 UTC m=+2249.136111499" watchObservedRunningTime="2026-11-04 01:34:40.378609438 +0000 UTC m=+2249.137271599" Nov 04 01:34:40.642381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:40.642341 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:40.642381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:40.642376 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:41.273532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:41.273496 4486 generic.go:356] "Generic (PLEG): container finished" podID="a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" containerID="2190e660987deb48f6abdaee215aeee8e4842976fdbad6a99810ba5fb11b5f46" exitCode=0 Nov 04 01:34:41.273951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:41.273577 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" event={"ID":"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0","Type":"ContainerDied","Data":"2190e660987deb48f6abdaee215aeee8e4842976fdbad6a99810ba5fb11b5f46"} Nov 04 01:34:42.420940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.420909 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:42.450176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.450138 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\" (UniqueName: \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\") pod \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " Nov 04 01:34:42.450376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.450226 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\") pod \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " Nov 04 01:34:42.450376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.450287 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\") pod \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\" (UID: \"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0\") " Nov 04 01:34:42.450650 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.450617 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca" pod "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" (UID: "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:42.452443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.452410 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca" pod "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" (UID: "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:42.452576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.452539 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m" pod "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" (UID: "a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0"). InnerVolumeSpecName "kube-api-access-gtj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:42.551943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.551851 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtj2m\" (UniqueName: \"kubernetes.io/projected/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-kube-api-access-gtj2m\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.551943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.551883 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.551943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.551897 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.640154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.640077 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" containerID="cri-o://1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599" gracePeriod=45 Nov 04 01:34:42.893972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.893945 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:34:42.958000 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.957960 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958030 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958090 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\" (UniqueName: \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958207 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958121 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958247 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.958454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958314 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958392 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\") pod \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\" (UID: \"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2\") " Nov 04 01:34:42.958701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958393 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.958775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958637 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:42.958913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958892 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.958976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958920 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.958976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.958936 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.960480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.960456 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "kube-api-access-pfrmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:42.960624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.960474 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:42.960624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:42.960542 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm" pod "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" (UID: "a03ef8e4-16eb-481f-af79-9afcf6ac7ac2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:43.060370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.060333 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.060370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.060365 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.060370 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.060376 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfrmn\" (UniqueName: \"kubernetes.io/projected/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2-kube-api-access-pfrmn\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:43.290975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.290891 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" Nov 04 01:34:43.290975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.290909 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f" event={"ID":"a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0","Type":"ContainerDied","Data":"9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c"} Nov 04 01:34:43.290975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.290969 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9def31e2f416cd71005b1f5292f1e815fc1736d9910ee5d233e02f30af3b722c" Nov 04 01:34:43.292654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.292622 4486 generic.go:356] "Generic (PLEG): container finished" podID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerID="1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599" exitCode=0 Nov 04 01:34:43.292821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.292656 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" event={"ID":"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2","Type":"ContainerDied","Data":"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599"} Nov 04 01:34:43.292821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.292681 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" Nov 04 01:34:43.292821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.292715 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv" event={"ID":"a03ef8e4-16eb-481f-af79-9afcf6ac7ac2","Type":"ContainerDied","Data":"2c2d5de3ca78172f61f15f23406f46a63831ec31a829a8aecaca4d5cdbf6081c"} Nov 04 01:34:43.292821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.292740 4486 scope.go:122] "RemoveContainer" containerID="1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599" Nov 04 01:34:43.305571 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.305546 4486 scope.go:122] "RemoveContainer" containerID="1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599" Nov 04 01:34:43.305905 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:43.305882 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599\": container with ID starting with 1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599 not found: ID does not exist" containerID="1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599" Nov 04 01:34:43.306010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.305917 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599"} err="failed to get container status \"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599\": rpc error: code = NotFound desc = could not find container \"1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599\": container with ID starting with 1c38db599f5dba8527b4f5f22bc78717b4d131ad2d9cf84d8316ccf17c546599 not found: ID does not exist" Nov 04 01:34:43.523274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.523234 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:34:43.529304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:43.529269 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2fd27d/distinct-w2-2fd27d-kserve-8cddb5d74-hxtzv"] Nov 04 01:34:43.833886 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:34:43.833838 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:34:45.349072 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:45.349029 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" path="/var/lib/kubelet/pods/a03ef8e4-16eb-481f-af79-9afcf6ac7ac2/volumes" Nov 04 01:34:50.642262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:50.642209 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:34:50.642779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:34:50.642204 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:35:00.103732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.103663 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:35:00.215041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.215006 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:35:00.274706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.274658 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\" (UniqueName: \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\") pod \"4846dc50-4398-4e49-8d6b-725a49dc0007\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " Nov 04 01:35:00.274910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.274789 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\") pod \"4846dc50-4398-4e49-8d6b-725a49dc0007\" (UID: \"4846dc50-4398-4e49-8d6b-725a49dc0007\") " Nov 04 01:35:00.275255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.275221 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume" pod "4846dc50-4398-4e49-8d6b-725a49dc0007" (UID: "4846dc50-4398-4e49-8d6b-725a49dc0007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:00.276935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.276907 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88" pod "4846dc50-4398-4e49-8d6b-725a49dc0007" (UID: "4846dc50-4398-4e49-8d6b-725a49dc0007"). InnerVolumeSpecName "kube-api-access-4xh88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:00.375617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.375534 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\" (UniqueName: \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\") pod \"dd726207-bfe1-42c9-a2ae-8b90e9084082\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " Nov 04 01:35:00.375779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.375678 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\") pod \"dd726207-bfe1-42c9-a2ae-8b90e9084082\" (UID: \"dd726207-bfe1-42c9-a2ae-8b90e9084082\") " Nov 04 01:35:00.375926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.375911 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xh88\" (UniqueName: \"kubernetes.io/projected/4846dc50-4398-4e49-8d6b-725a49dc0007-kube-api-access-4xh88\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.376021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.375928 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4846dc50-4398-4e49-8d6b-725a49dc0007-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.376089 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.376063 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume" pod "dd726207-bfe1-42c9-a2ae-8b90e9084082" (UID: "dd726207-bfe1-42c9-a2ae-8b90e9084082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:00.377669 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.377646 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx" pod "dd726207-bfe1-42c9-a2ae-8b90e9084082" (UID: "dd726207-bfe1-42c9-a2ae-8b90e9084082"). InnerVolumeSpecName "kube-api-access-b44hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:00.419691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.419653 4486 generic.go:356] "Generic (PLEG): container finished" podID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerID="a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c" exitCode=0 Nov 04 01:35:00.419876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.419701 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" Nov 04 01:35:00.419876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.419749 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" event={"ID":"dd726207-bfe1-42c9-a2ae-8b90e9084082","Type":"ContainerDied","Data":"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c"} Nov 04 01:35:00.419876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.419807 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72" event={"ID":"dd726207-bfe1-42c9-a2ae-8b90e9084082","Type":"ContainerDied","Data":"d84ae038368cc44a7eca9435cd94186ca0b84fce8ba4d45b02d051b42c1d7beb"} Nov 04 01:35:00.419876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.419834 4486 scope.go:122] "RemoveContainer" containerID="a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c" Nov 04 01:35:00.421509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.421480 4486 generic.go:356] "Generic (PLEG): container finished" podID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerID="4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d" exitCode=0 Nov 04 01:35:00.421656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.421517 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" Nov 04 01:35:00.421656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.421543 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" event={"ID":"4846dc50-4398-4e49-8d6b-725a49dc0007","Type":"ContainerDied","Data":"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d"} Nov 04 01:35:00.421656 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.421585 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl" event={"ID":"4846dc50-4398-4e49-8d6b-725a49dc0007","Type":"ContainerDied","Data":"c5ce6da38ced974bf010a3371ab2d462972e2cae8549e5c878dc4e5d35834a91"} Nov 04 01:35:00.431869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.431841 4486 scope.go:122] "RemoveContainer" containerID="a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c" Nov 04 01:35:00.432212 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:00.432190 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c\": container with ID starting with a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c not found: ID does not exist" containerID="a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c" Nov 04 01:35:00.432320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.432221 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c"} err="failed to get container status \"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c\": rpc error: code = NotFound desc = could not find container \"a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c\": container with ID starting with a7f52040baab93759367df04c58ae26777c8c926e1721f8e088cb74a8703f69c not found: ID does not exist" Nov 04 01:35:00.432320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.432241 4486 scope.go:122] "RemoveContainer" containerID="4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d" Nov 04 01:35:00.445421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.445393 4486 scope.go:122] "RemoveContainer" containerID="4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d" Nov 04 01:35:00.445822 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:00.445782 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d\": container with ID starting with 4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d not found: ID does not exist" containerID="4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d" Nov 04 01:35:00.445951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.445832 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d"} err="failed to get container status \"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d\": rpc error: code = NotFound desc = could not find container \"4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d\": container with ID starting with 4261f627f410bf2a5ad33678643101f8aa34f52643328b8ff7507a7e78860e8d not found: ID does not exist" Nov 04 01:35:00.477186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.477145 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44hx\" (UniqueName: \"kubernetes.io/projected/dd726207-bfe1-42c9-a2ae-8b90e9084082-kube-api-access-b44hx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.477186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.477185 4486 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd726207-bfe1-42c9-a2ae-8b90e9084082-plugins-config-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.577581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.577541 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:35:00.584765 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.584724 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2fd27d-696f95f848-qnfnl"] Nov 04 01:35:00.690762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.690676 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:35:00.698063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:00.698027 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2fd27d-5b67b7d555-xrt72"] Nov 04 01:35:01.069404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.069365 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c"] Nov 04 01:35:01.073412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.073381 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf"] Nov 04 01:35:01.077186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.077150 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b8051b-x9jvf"] Nov 04 01:35:01.116654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.116616 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87d9f19a-xzl2c"] Nov 04 01:35:01.362853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.362761 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5221c9-adf8-49d3-90eb-ca45d2bc17b7" path="/var/lib/kubelet/pods/2e5221c9-adf8-49d3-90eb-ca45d2bc17b7/volumes" Nov 04 01:35:01.363368 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.363336 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4731bd77-09b8-4ab5-b439-8670affb5d57" path="/var/lib/kubelet/pods/4731bd77-09b8-4ab5-b439-8670affb5d57/volumes" Nov 04 01:35:01.363796 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.363777 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" path="/var/lib/kubelet/pods/4846dc50-4398-4e49-8d6b-725a49dc0007/volumes" Nov 04 01:35:01.364680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:01.364657 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" path="/var/lib/kubelet/pods/dd726207-bfe1-42c9-a2ae-8b90e9084082/volumes" Nov 04 01:35:09.661668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.661620 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:35:09.662090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.661912 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-566cd577c-lz4fn" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" containerID="cri-o://b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b" gracePeriod=30 Nov 04 01:35:09.962751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.962723 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:35:09.973915 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.973885 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\") pod \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " Nov 04 01:35:09.974085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.973936 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\") pod \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " Nov 04 01:35:09.974085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.974032 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\" (UniqueName: \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\") pod \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\" (UID: \"af9b71b3-2696-4104-b71d-f2b12dfef6a4\") " Nov 04 01:35:09.976068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.976026 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert" pod "af9b71b3-2696-4104-b71d-f2b12dfef6a4" (UID: "af9b71b3-2696-4104-b71d-f2b12dfef6a4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:09.976206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.976128 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w" pod "af9b71b3-2696-4104-b71d-f2b12dfef6a4" (UID: "af9b71b3-2696-4104-b71d-f2b12dfef6a4"). InnerVolumeSpecName "kube-api-access-ckk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:09.976206 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:09.976157 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs" pod "af9b71b3-2696-4104-b71d-f2b12dfef6a4" (UID: "af9b71b3-2696-4104-b71d-f2b12dfef6a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:10.075412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.075374 4486 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-metrics-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.075412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.075406 4486 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af9b71b3-2696-4104-b71d-f2b12dfef6a4-webhook-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.075412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.075417 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckk8w\" (UniqueName: \"kubernetes.io/projected/af9b71b3-2696-4104-b71d-f2b12dfef6a4-kube-api-access-ckk8w\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.497964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.497929 4486 generic.go:356] "Generic (PLEG): container finished" podID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerID="b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b" exitCode=0 Nov 04 01:35:10.498338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.498024 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-566cd577c-lz4fn" Nov 04 01:35:10.498338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.498030 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerDied","Data":"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b"} Nov 04 01:35:10.498338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.498090 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-566cd577c-lz4fn" event={"ID":"af9b71b3-2696-4104-b71d-f2b12dfef6a4","Type":"ContainerDied","Data":"1ce6ff8a3cbc3ccd29fe636b28772f752b6c31ec5cb2c904dd72cdce73a73723"} Nov 04 01:35:10.498338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.498114 4486 scope.go:122] "RemoveContainer" containerID="b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b" Nov 04 01:35:10.510353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.510328 4486 scope.go:122] "RemoveContainer" containerID="e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c" Nov 04 01:35:10.520289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.520265 4486 scope.go:122] "RemoveContainer" containerID="b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b" Nov 04 01:35:10.520606 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:10.520584 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b\": container with ID starting with b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b not found: ID does not exist" containerID="b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b" Nov 04 01:35:10.520673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.520616 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b"} err="failed to get container status \"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b\": rpc error: code = NotFound desc = could not find container \"b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b\": container with ID starting with b5fc86755adcaa3a06e42b6f4bc8f3bcd03a04194e15c767d60de64a3d678b4b not found: ID does not exist" Nov 04 01:35:10.520673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.520634 4486 scope.go:122] "RemoveContainer" containerID="e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c" Nov 04 01:35:10.520913 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:10.520895 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c\": container with ID starting with e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c not found: ID does not exist" containerID="e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c" Nov 04 01:35:10.520961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.520920 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c"} err="failed to get container status \"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c\": rpc error: code = NotFound desc = could not find container \"e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c\": container with ID starting with e420e82e4edfdf61a8b61a4425cc87e2c384d54ba729f811f57a41a048f1cb4c not found: ID does not exist" Nov 04 01:35:10.659584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.659510 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:35:10.706909 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:10.706862 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-566cd577c-lz4fn"] Nov 04 01:35:11.348902 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:11.348869 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" path="/var/lib/kubelet/pods/af9b71b3-2696-4104-b71d-f2b12dfef6a4/volumes" Nov 04 01:35:13.831701 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:13.831659 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:35:20.563805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:20.563770 4486 scope.go:122] "RemoveContainer" containerID="3e5cc98b46ce1f8f4d6b9590f36cf06ba30950a0aa362ebd2897532ff959cfe7" Nov 04 01:35:20.575328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:20.575303 4486 scope.go:122] "RemoveContainer" containerID="e8fb46f0fdab6453669174a97bb19f5533cc451b34aa4ca2e3b2245aad224dd4" Nov 04 01:35:20.588019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:35:20.587994 4486 scope.go:122] "RemoveContainer" containerID="ce916aca4142a3077a60c5431f9c6c192e23245e4cb83a12d2d3aae1a94f8f04" Nov 04 01:35:43.831636 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:35:43.831592 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:36:13.831551 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:36:13.831510 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:36:20.673025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:20.672972 4486 scope.go:122] "RemoveContainer" containerID="e8a4b87b0d2e1110ab9542c7130a4027b32eabba091150db2999a8fa02bff608" Nov 04 01:36:20.682661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:20.682640 4486 scope.go:122] "RemoveContainer" containerID="d8d1c27284326041055dea58f9a93b009ae0af6451704860a9df96c77bfaeece" Nov 04 01:36:20.692379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:20.692357 4486 scope.go:122] "RemoveContainer" containerID="def09d85df9c38542696d36618cf2aa4317ea37ab86486440f4ff74d9a17747d" Nov 04 01:36:39.863289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:39.862138 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:36:39.863960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:39.863905 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:36:43.831269 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:36:43.831228 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:36:44.370612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.370574 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:36:44.370824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.370790 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerName="discovery" containerID="cri-o://d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918" gracePeriod=30 Nov 04 01:36:44.629274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.629210 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:36:44.640255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640223 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640269 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640303 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640335 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640392 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640360 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640436 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640509 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\") pod \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\" (UID: \"2111fa87-0b9d-47cf-b1de-f5406ca8fd18\") " Nov 04 01:36:44.640789 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640750 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:44.640940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.640918 4486 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-ca-configmap\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.642680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.642647 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:44.642827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.642799 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:44.643067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.643017 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:44.643161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.643115 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:44.643257 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.643237 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "kube-api-access-sn6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:44.643321 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.643254 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token" pod "2111fa87-0b9d-47cf-b1de-f5406ca8fd18" (UID: "2111fa87-0b9d-47cf-b1de-f5406ca8fd18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:44.741846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741789 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6cd\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-kube-api-access-sn6cd\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.741846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741836 4486 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-cacerts\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.741846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741851 4486 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-local-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.742286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741864 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.742286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741877 4486 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-kubeconfig\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:44.742286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:44.741889 4486 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2111fa87-0b9d-47cf-b1de-f5406ca8fd18-istio-csr-dns-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:45.245097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.245050 4486 generic.go:356] "Generic (PLEG): container finished" podID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerID="d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918" exitCode=0 Nov 04 01:36:45.245097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.245090 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" Nov 04 01:36:45.245709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.245123 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" event={"ID":"2111fa87-0b9d-47cf-b1de-f5406ca8fd18","Type":"ContainerDied","Data":"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918"} Nov 04 01:36:45.245709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.245181 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh" event={"ID":"2111fa87-0b9d-47cf-b1de-f5406ca8fd18","Type":"ContainerDied","Data":"9ebb5f6193d92ea245269afa59fdcf473e3e8cb763fc55fc58e0ac011f4041dc"} Nov 04 01:36:45.245709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.245202 4486 scope.go:122] "RemoveContainer" containerID="d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918" Nov 04 01:36:45.257212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.257189 4486 scope.go:122] "RemoveContainer" containerID="d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918" Nov 04 01:36:45.257547 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:36:45.257526 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918\": container with ID starting with d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918 not found: ID does not exist" containerID="d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918" Nov 04 01:36:45.257643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.257556 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918"} err="failed to get container status \"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918\": rpc error: code = NotFound desc = could not find container \"d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918\": container with ID starting with d1741b83aa05e34afbcfb124dd74e4d5a1a5d2c82714dae3173a823c68a1a918 not found: ID does not exist" Nov 04 01:36:45.453500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.453451 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:36:45.459329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.459293 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fn8kh"] Nov 04 01:36:45.864866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.864823 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2"] Nov 04 01:36:45.865352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865338 4486 cpu_manager.go:414] "containerMap: removing container" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865354 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865366 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865373 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3af9229-7b9f-4553-8896-b66f6a75b146" containerName="istio-proxy" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865395 4486 cpu_manager.go:414] "containerMap: removing container" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865400 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a87968-6e6d-4188-a8d5-1d03cc748fba" containerName="istio-proxy" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865409 4486 cpu_manager.go:414] "containerMap: removing container" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" Nov 04 01:36:45.865418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865420 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865430 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" containerName="revoke-keys" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865435 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" containerName="revoke-keys" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865442 4486 cpu_manager.go:414] "containerMap: removing container" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865448 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865460 4486 cpu_manager.go:414] "containerMap: removing container" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerName="discovery" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865469 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerName="discovery" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865484 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865488 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865502 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865506 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865567 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865577 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" containerName="discovery" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865586 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd726207-bfe1-42c9-a2ae-8b90e9084082" containerName="payload-pre-processing" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865594 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03ef8e4-16eb-481f-af79-9afcf6ac7ac2" containerName="main" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865602 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="4846dc50-4398-4e49-8d6b-725a49dc0007" containerName="payload-processing" Nov 04 01:36:45.865798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.865608 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" containerName="revoke-keys" Nov 04 01:36:45.870437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.870416 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:45.923127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.923090 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2"] Nov 04 01:36:45.954304 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.954261 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:45.954514 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.954483 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77pc\" (UniqueName: \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:45.954596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.954556 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:45.999242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.999204 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:36:45.999439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:45.999252 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:36:46.055351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.055237 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.055351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.055303 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.055634 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.055399 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77pc\" (UniqueName: \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.056130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.056103 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.057745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.057720 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.110730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.110693 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77pc\" (UniqueName: \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\") pod \"maas-api-revoke-sub-c1e271a9-454k2\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.185448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.185362 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:46.556465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:46.556432 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2"] Nov 04 01:36:46.558545 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:36:46.558517 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d4b781_b199_41a6_9f2b_90261b175140.slice/crio-20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839 WatchSource:0}: Error finding container 20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839: Status 404 returned error can't find the container with id 20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839 Nov 04 01:36:47.268632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:47.268536 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" event={"ID":"64d4b781-b199-41a6-9f2b-90261b175140","Type":"ContainerStarted","Data":"809fca64729bc1fe2e81ddaf48f86f427aa595e698f31382a868977e48952285"} Nov 04 01:36:47.268632 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:47.268592 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" event={"ID":"64d4b781-b199-41a6-9f2b-90261b175140","Type":"ContainerStarted","Data":"20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839"} Nov 04 01:36:47.347856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:47.347821 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2111fa87-0b9d-47cf-b1de-f5406ca8fd18" path="/var/lib/kubelet/pods/2111fa87-0b9d-47cf-b1de-f5406ca8fd18/volumes" Nov 04 01:36:47.379890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:47.379837 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" podStartSLOduration=2.379822375 podStartE2EDuration="2.379822375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:36:47.377628739 +0000 UTC m=+2376.136290901" watchObservedRunningTime="2026-11-04 01:36:47.379822375 +0000 UTC m=+2376.138484547" Nov 04 01:36:48.275999 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:48.275944 4486 generic.go:356] "Generic (PLEG): container finished" podID="64d4b781-b199-41a6-9f2b-90261b175140" containerID="809fca64729bc1fe2e81ddaf48f86f427aa595e698f31382a868977e48952285" exitCode=0 Nov 04 01:36:48.276428 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:48.276025 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" event={"ID":"64d4b781-b199-41a6-9f2b-90261b175140","Type":"ContainerDied","Data":"809fca64729bc1fe2e81ddaf48f86f427aa595e698f31382a868977e48952285"} Nov 04 01:36:49.426922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.426892 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:49.486155 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.486121 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\" (UniqueName: \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\") pod \"64d4b781-b199-41a6-9f2b-90261b175140\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " Nov 04 01:36:49.486334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.486166 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\") pod \"64d4b781-b199-41a6-9f2b-90261b175140\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " Nov 04 01:36:49.486334 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.486197 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\") pod \"64d4b781-b199-41a6-9f2b-90261b175140\" (UID: \"64d4b781-b199-41a6-9f2b-90261b175140\") " Nov 04 01:36:49.486639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.486608 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca" pod "64d4b781-b199-41a6-9f2b-90261b175140" (UID: "64d4b781-b199-41a6-9f2b-90261b175140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:49.488323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.488299 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc" pod "64d4b781-b199-41a6-9f2b-90261b175140" (UID: "64d4b781-b199-41a6-9f2b-90261b175140"). InnerVolumeSpecName "kube-api-access-t77pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:49.488448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.488352 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca" pod "64d4b781-b199-41a6-9f2b-90261b175140" (UID: "64d4b781-b199-41a6-9f2b-90261b175140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:36:49.587422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.587383 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77pc\" (UniqueName: \"kubernetes.io/projected/64d4b781-b199-41a6-9f2b-90261b175140-kube-api-access-t77pc\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:49.587422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.587419 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:49.587618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:49.587436 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d4b781-b199-41a6-9f2b-90261b175140-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:50.289729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:50.289701 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" Nov 04 01:36:50.289925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:50.289739 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2" event={"ID":"64d4b781-b199-41a6-9f2b-90261b175140","Type":"ContainerDied","Data":"20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839"} Nov 04 01:36:50.289925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:36:50.289787 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20bf5eb192848743786b46f2539d7e3019d36dad829a131e762163a7e6709839" Nov 04 01:37:12.044741 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:12.044698 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9"] Nov 04 01:37:12.058734 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:12.058697 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6eb7bf4-lwvp9"] Nov 04 01:37:13.047998 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:13.047952 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr"] Nov 04 01:37:13.095385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:13.095345 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d9d449-crptr"] Nov 04 01:37:13.348317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:13.348277 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a787480-7a17-4ed8-8270-48b80a6c23ce" path="/var/lib/kubelet/pods/4a787480-7a17-4ed8-8270-48b80a6c23ce/volumes" Nov 04 01:37:13.348824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:13.348804 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7273b770-32c1-44bc-8246-1dd76167a40c" path="/var/lib/kubelet/pods/7273b770-32c1-44bc-8246-1dd76167a40c/volumes" Nov 04 01:37:13.831349 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:37:13.831296 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:37:17.528212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.528172 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f"] Nov 04 01:37:17.528643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.528629 4486 cpu_manager.go:414] "containerMap: removing container" podUID="64d4b781-b199-41a6-9f2b-90261b175140" containerName="revoke-keys" Nov 04 01:37:17.528690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.528643 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d4b781-b199-41a6-9f2b-90261b175140" containerName="revoke-keys" Nov 04 01:37:17.528743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.528734 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9b71b3-2696-4104-b71d-f2b12dfef6a4" containerName="manager" Nov 04 01:37:17.528779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.528748 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d4b781-b199-41a6-9f2b-90261b175140" containerName="revoke-keys" Nov 04 01:37:17.533124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.533103 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.585807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.585754 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f"] Nov 04 01:37:17.601592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.601557 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjxl\" (UniqueName: \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.601774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.601613 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.601774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.601699 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.629465 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.629398 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:17.629680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.629659 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:37:17.702613 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.702573 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjxl\" (UniqueName: \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.702806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.702618 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.702806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.702647 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.703345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.703322 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.705079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.705058 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.757579 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.757540 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjxl\" (UniqueName: \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\") pod \"maas-api-revoke-sub-2e1f5b5f-5n26f\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:17.846291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:17.846251 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:18.002286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.002252 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:37:18.016900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.016871 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:37:18.034814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.034784 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:37:18.048004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.047954 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:37:18.216646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.216602 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f"] Nov 04 01:37:18.218089 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:37:18.218064 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ad6e08_b951_44f6_a7a3_6e8b53e5fdd7.slice/crio-a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139 WatchSource:0}: Error finding container a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139: Status 404 returned error can't find the container with id a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139 Nov 04 01:37:18.498821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:18.498729 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" event={"ID":"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7","Type":"ContainerStarted","Data":"a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139"} Nov 04 01:37:19.506376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:19.506265 4486 generic.go:356] "Generic (PLEG): container finished" podID="e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" containerID="4b4ff93872954b1d57f65e2345f9c2053b50e29f6399f9b7b0ac82eb11b4f37c" exitCode=0 Nov 04 01:37:19.506376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:19.506362 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" event={"ID":"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7","Type":"ContainerDied","Data":"4b4ff93872954b1d57f65e2345f9c2053b50e29f6399f9b7b0ac82eb11b4f37c"} Nov 04 01:37:20.644630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.644597 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:20.726923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.726887 4486 scope.go:122] "RemoveContainer" containerID="80d4977950af3a98890447d32c6fb6429be350acc03d5708848ca415609489fb" Nov 04 01:37:20.732200 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.732172 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\" (UniqueName: \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\") pod \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " Nov 04 01:37:20.732351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.732318 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\") pod \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " Nov 04 01:37:20.732419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.732359 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\") pod \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\" (UID: \"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7\") " Nov 04 01:37:20.732654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.732624 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca" pod "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" (UID: "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:20.732775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.732730 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:20.734374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.734343 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca" pod "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" (UID: "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:20.734374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.734353 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl" pod "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" (UID: "e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7"). InnerVolumeSpecName "kube-api-access-pmjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:20.736946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.736928 4486 scope.go:122] "RemoveContainer" containerID="ba5693b8bb1d583255c7254893aa5714f8b6d0db131aac98598defd74efe3e24" Nov 04 01:37:20.754412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.754383 4486 scope.go:122] "RemoveContainer" containerID="94443bccec55f4fa89f68fb9f7ef0ba3a3221b019ff27e719c7530e0f6ba9a26" Nov 04 01:37:20.833621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.833593 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:20.833621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:20.833621 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjxl\" (UniqueName: \"kubernetes.io/projected/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7-kube-api-access-pmjxl\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:21.519704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:21.519673 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" Nov 04 01:37:21.519890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:21.519710 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f" event={"ID":"e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7","Type":"ContainerDied","Data":"a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139"} Nov 04 01:37:21.519890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:21.519758 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47a878ce37fdba518f2aeb3a56406f1f701cedb8e65da384af40574f06da139" Nov 04 01:37:29.891197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.891159 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8"] Nov 04 01:37:29.891626 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.891613 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" containerName="revoke-keys" Nov 04 01:37:29.891672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.891627 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" containerName="revoke-keys" Nov 04 01:37:29.891706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.891702 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" containerName="revoke-keys" Nov 04 01:37:29.896816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.896789 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:29.948949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.948911 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8"] Nov 04 01:37:29.993230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.993194 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:37:29.993419 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:29.993245 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:37:30.028266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.028228 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.028455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.028296 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwd9\" (UniqueName: \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.028455 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.028438 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.129712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.129668 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.129896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.129755 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.129896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.129802 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwd9\" (UniqueName: \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.130338 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.130319 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.132179 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.132158 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.184953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.184865 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwd9\" (UniqueName: \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\") pod \"maas-api-revoke-sub-ec843018-vsvn8\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.211068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.211024 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:30.582965 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:37:30.582930 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5692ffc_8e81_4f80_a861_e05d46f3c1a4.slice/crio-8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc WatchSource:0}: Error finding container 8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc: Status 404 returned error can't find the container with id 8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc Nov 04 01:37:30.583125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:30.582964 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8"] Nov 04 01:37:31.586657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:31.586624 4486 generic.go:356] "Generic (PLEG): container finished" podID="b5692ffc-8e81-4f80-a861-e05d46f3c1a4" containerID="d124df22263459c7b12adddce1b92ef077ebca0263611b21514fa8246bae1db1" exitCode=0 Nov 04 01:37:31.587135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:31.586716 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" event={"ID":"b5692ffc-8e81-4f80-a861-e05d46f3c1a4","Type":"ContainerDied","Data":"d124df22263459c7b12adddce1b92ef077ebca0263611b21514fa8246bae1db1"} Nov 04 01:37:31.587135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:31.586782 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" event={"ID":"b5692ffc-8e81-4f80-a861-e05d46f3c1a4","Type":"ContainerStarted","Data":"8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc"} Nov 04 01:37:32.734317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.734285 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:32.861311 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.861216 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\") pod \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " Nov 04 01:37:32.863488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.861308 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\") pod \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " Nov 04 01:37:32.863488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.861430 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\" (UniqueName: \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\") pod \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\" (UID: \"b5692ffc-8e81-4f80-a861-e05d46f3c1a4\") " Nov 04 01:37:32.863488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.861650 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca" pod "b5692ffc-8e81-4f80-a861-e05d46f3c1a4" (UID: "b5692ffc-8e81-4f80-a861-e05d46f3c1a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:37:32.863488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.861804 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:32.863680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.863487 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca" pod "b5692ffc-8e81-4f80-a861-e05d46f3c1a4" (UID: "b5692ffc-8e81-4f80-a861-e05d46f3c1a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:37:32.863680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.863544 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9" pod "b5692ffc-8e81-4f80-a861-e05d46f3c1a4" (UID: "b5692ffc-8e81-4f80-a861-e05d46f3c1a4"). InnerVolumeSpecName "kube-api-access-hxwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:37:32.962745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.962710 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:32.962745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:32.962742 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwd9\" (UniqueName: \"kubernetes.io/projected/b5692ffc-8e81-4f80-a861-e05d46f3c1a4-kube-api-access-hxwd9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:37:33.604356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:33.604311 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" Nov 04 01:37:33.604356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:33.604329 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8" event={"ID":"b5692ffc-8e81-4f80-a861-e05d46f3c1a4","Type":"ContainerDied","Data":"8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc"} Nov 04 01:37:33.604584 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:33.604389 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2e05ffe53b7f69fc0c433c031fb3d2ba1df42126ef7ce16925af62eee2e3cc" Nov 04 01:37:43.832041 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:37:43.831995 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:37:49.061008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.060503 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh"] Nov 04 01:37:49.100937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.100891 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb77332-qdrrh"] Nov 04 01:37:49.104255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.104220 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr"] Nov 04 01:37:49.107126 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.107092 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4febda-gn6rr"] Nov 04 01:37:49.388009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.387309 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38bb5d23-ea17-4913-ac6d-d5737f32b038" path="/var/lib/kubelet/pods/38bb5d23-ea17-4913-ac6d-d5737f32b038/volumes" Nov 04 01:37:49.388255 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:37:49.388158 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd3fdd7-9484-4527-b713-a043371a9da9" path="/var/lib/kubelet/pods/9dd3fdd7-9484-4527-b713-a043371a9da9/volumes" Nov 04 01:38:05.571387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.571350 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn"] Nov 04 01:38:05.571793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.571780 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b5692ffc-8e81-4f80-a861-e05d46f3c1a4" containerName="revoke-keys" Nov 04 01:38:05.571840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.571794 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5692ffc-8e81-4f80-a861-e05d46f3c1a4" containerName="revoke-keys" Nov 04 01:38:05.571897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.571887 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5692ffc-8e81-4f80-a861-e05d46f3c1a4" containerName="revoke-keys" Nov 04 01:38:05.576581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.576560 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.628502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.628468 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn"] Nov 04 01:38:05.647840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.647804 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.648081 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.647857 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.648175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.648059 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqg4b\" (UniqueName: \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.672076 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.672041 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:05.672212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.672089 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:38:05.749125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.749087 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.749337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.749139 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.749337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.749207 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqg4b\" (UniqueName: \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.749938 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.749909 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.751621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.751599 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.808657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.808623 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqg4b\" (UniqueName: \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\") pod \"maas-api-revoke-sub-046e1343-nzvgn\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:05.889403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:05.889320 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:06.262548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:06.262517 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn"] Nov 04 01:38:06.264493 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:38:06.264463 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f147fd6_d73d_4b0e_a1fb_b7eced7cf6fc.slice/crio-2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2 WatchSource:0}: Error finding container 2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2: Status 404 returned error can't find the container with id 2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2 Nov 04 01:38:06.818191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:06.818157 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" event={"ID":"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc","Type":"ContainerStarted","Data":"2caa93454aef36f28c09490050b60342e0284eed043e6f2b02f38ab48ed273bc"} Nov 04 01:38:06.818596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:06.818213 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" event={"ID":"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc","Type":"ContainerStarted","Data":"2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2"} Nov 04 01:38:06.930940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:06.930881 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" podStartSLOduration=1.930867694 podStartE2EDuration="1.930867694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:38:06.928796708 +0000 UTC m=+2455.687458872" watchObservedRunningTime="2026-11-04 01:38:06.930867694 +0000 UTC m=+2455.689529852" Nov 04 01:38:07.828110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:07.828065 4486 generic.go:356] "Generic (PLEG): container finished" podID="7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" containerID="2caa93454aef36f28c09490050b60342e0284eed043e6f2b02f38ab48ed273bc" exitCode=0 Nov 04 01:38:07.828539 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:07.828141 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" event={"ID":"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc","Type":"ContainerDied","Data":"2caa93454aef36f28c09490050b60342e0284eed043e6f2b02f38ab48ed273bc"} Nov 04 01:38:08.978083 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:08.978054 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:09.085235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.085193 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\" (UniqueName: \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\") pod \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " Nov 04 01:38:09.085494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.085255 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\") pod \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " Nov 04 01:38:09.085494 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.085287 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\") pod \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\" (UID: \"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc\") " Nov 04 01:38:09.085646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.085622 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca" pod "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" (UID: "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:09.087795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.087743 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca" pod "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" (UID: "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:09.088015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.087830 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b" pod "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" (UID: "7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc"). InnerVolumeSpecName "kube-api-access-kqg4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:09.186601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.186555 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqg4b\" (UniqueName: \"kubernetes.io/projected/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-kube-api-access-kqg4b\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:09.186601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.186592 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:09.186601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.186603 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:09.844048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.844020 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" Nov 04 01:38:09.844274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.844014 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn" event={"ID":"7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc","Type":"ContainerDied","Data":"2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2"} Nov 04 01:38:09.844274 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:09.844141 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f28ffa59e4b25c4429611b69de6e6e4dc8b5f28b8a32cfb07a1663dbc3f6bf2" Nov 04 01:38:13.831169 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:38:13.831129 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:38:16.334560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.334521 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw"] Nov 04 01:38:16.335276 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.335254 4486 cpu_manager.go:414] "containerMap: removing container" podUID="7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" containerName="revoke-keys" Nov 04 01:38:16.335330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.335279 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" containerName="revoke-keys" Nov 04 01:38:16.335401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.335390 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" containerName="revoke-keys" Nov 04 01:38:16.339305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.339281 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.391723 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.391685 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw"] Nov 04 01:38:16.434876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.434844 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:16.435067 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.434937 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:38:16.458138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.458103 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.458324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.458171 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.458324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.458295 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jk8\" (UniqueName: \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.559752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.559719 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.559960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.559784 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.559960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.559864 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jk8\" (UniqueName: \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.560509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.560482 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.562383 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.562360 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.614567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.614484 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jk8\" (UniqueName: \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\") pod \"maas-api-revoke-sub-d3fba056-sm8qw\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:16.656994 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:16.656940 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:17.031666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:17.031631 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw"] Nov 04 01:38:17.032922 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:38:17.032891 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c17621_85f9_4246_860f_5b9c15d604e8.slice/crio-339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde WatchSource:0}: Error finding container 339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde: Status 404 returned error can't find the container with id 339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde Nov 04 01:38:17.907033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:17.906971 4486 generic.go:356] "Generic (PLEG): container finished" podID="e9c17621-85f9-4246-860f-5b9c15d604e8" containerID="021bba3bd887e42a1bd282dca824e10f13caa0bac0963c036fe7ae12370a60d9" exitCode=0 Nov 04 01:38:17.907409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:17.907047 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" event={"ID":"e9c17621-85f9-4246-860f-5b9c15d604e8","Type":"ContainerDied","Data":"021bba3bd887e42a1bd282dca824e10f13caa0bac0963c036fe7ae12370a60d9"} Nov 04 01:38:17.907409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:17.907133 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" event={"ID":"e9c17621-85f9-4246-860f-5b9c15d604e8","Type":"ContainerStarted","Data":"339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde"} Nov 04 01:38:19.054013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.053965 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:19.189409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.189316 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\" (UniqueName: \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\") pod \"e9c17621-85f9-4246-860f-5b9c15d604e8\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " Nov 04 01:38:19.189409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.189393 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\") pod \"e9c17621-85f9-4246-860f-5b9c15d604e8\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " Nov 04 01:38:19.189659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.189428 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\") pod \"e9c17621-85f9-4246-860f-5b9c15d604e8\" (UID: \"e9c17621-85f9-4246-860f-5b9c15d604e8\") " Nov 04 01:38:19.189951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.189922 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca" pod "e9c17621-85f9-4246-860f-5b9c15d604e8" (UID: "e9c17621-85f9-4246-860f-5b9c15d604e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:19.193538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.193506 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8" pod "e9c17621-85f9-4246-860f-5b9c15d604e8" (UID: "e9c17621-85f9-4246-860f-5b9c15d604e8"). InnerVolumeSpecName "kube-api-access-t2jk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:19.195142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.195111 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca" pod "e9c17621-85f9-4246-860f-5b9c15d604e8" (UID: "e9c17621-85f9-4246-860f-5b9c15d604e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:19.290944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.290897 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jk8\" (UniqueName: \"kubernetes.io/projected/e9c17621-85f9-4246-860f-5b9c15d604e8-kube-api-access-t2jk8\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:19.290944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.290937 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:19.290944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.290953 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c17621-85f9-4246-860f-5b9c15d604e8-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:19.927480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.927441 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" Nov 04 01:38:19.927732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.927472 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw" event={"ID":"e9c17621-85f9-4246-860f-5b9c15d604e8","Type":"ContainerDied","Data":"339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde"} Nov 04 01:38:19.927732 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:19.927547 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339dac7f359d0b0f3b825b2aef21cfabcfc5e7cad182e47432d6fc4c47954bde" Nov 04 01:38:20.841189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:20.841032 4486 scope.go:122] "RemoveContainer" containerID="77e20944ea31a486e54b10d8eb2444f5cfeef2cbdd343055d1c6fc5bbad54f06" Nov 04 01:38:20.907086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:20.902216 4486 scope.go:122] "RemoveContainer" containerID="e0d260dc5f20209bc5880fa54e7fa71ac6b22fa01bf5171a905d73568f0f1c4c" Nov 04 01:38:20.920365 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:20.920334 4486 scope.go:122] "RemoveContainer" containerID="cbc5cda7547c97c4e4c5368fdccf39ca2f8a9414573aa19661b01900fda68886" Nov 04 01:38:20.930926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:20.930825 4486 scope.go:122] "RemoveContainer" containerID="8eab173a62c2a5341af5f4c68de39f576f459477af80253a1cb5f09f0647bd76" Nov 04 01:38:43.831545 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:38:43.831507 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:38:51.317227 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.317184 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54"] Nov 04 01:38:51.317946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.317857 4486 cpu_manager.go:414] "containerMap: removing container" podUID="e9c17621-85f9-4246-860f-5b9c15d604e8" containerName="revoke-keys" Nov 04 01:38:51.317946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.317878 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c17621-85f9-4246-860f-5b9c15d604e8" containerName="revoke-keys" Nov 04 01:38:51.318116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.318051 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c17621-85f9-4246-860f-5b9c15d604e8" containerName="revoke-keys" Nov 04 01:38:51.321603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.321577 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.352012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.351951 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.352233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.352078 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.352233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.352209 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqbxg\" (UniqueName: \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.375209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.375179 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54"] Nov 04 01:38:51.419115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.419082 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:51.419115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.419091 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:38:51.453139 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.453102 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqbxg\" (UniqueName: \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.453343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.453191 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.453473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.453443 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.454086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.454063 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.455684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.455647 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.508191 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.508153 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqbxg\" (UniqueName: \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\") pod \"maas-api-revoke-sub-ae594979-mdm54\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:51.636253 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:51.636157 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:52.015298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:52.007162 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54"] Nov 04 01:38:52.191795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:52.191745 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" event={"ID":"c938248d-ce29-4b56-98f7-5a2eba02226f","Type":"ContainerStarted","Data":"c8b174ccb5c6b956cce1343810f4157c0864b7f98865dd37ca85b58a272cee88"} Nov 04 01:38:53.053242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:53.053202 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq"] Nov 04 01:38:53.100696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:53.100649 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524a85d0-nf8zq"] Nov 04 01:38:53.199429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:53.199340 4486 generic.go:356] "Generic (PLEG): container finished" podID="c938248d-ce29-4b56-98f7-5a2eba02226f" containerID="70d1ff99d71700ce3d66a5e5991d1a9cdcd39c0c7316c7e9ae3768678b160f46" exitCode=0 Nov 04 01:38:53.199590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:53.199429 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" event={"ID":"c938248d-ce29-4b56-98f7-5a2eba02226f","Type":"ContainerDied","Data":"70d1ff99d71700ce3d66a5e5991d1a9cdcd39c0c7316c7e9ae3768678b160f46"} Nov 04 01:38:53.348041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:53.347978 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbde716-c40d-4ef9-8d20-32f72cd651bc" path="/var/lib/kubelet/pods/3fbde716-c40d-4ef9-8d20-32f72cd651bc/volumes" Nov 04 01:38:54.357698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.357672 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:54.385567 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.385510 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\") pod \"c938248d-ce29-4b56-98f7-5a2eba02226f\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " Nov 04 01:38:54.385762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.385601 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\") pod \"c938248d-ce29-4b56-98f7-5a2eba02226f\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " Nov 04 01:38:54.385762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.385627 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\" (UniqueName: \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\") pod \"c938248d-ce29-4b56-98f7-5a2eba02226f\" (UID: \"c938248d-ce29-4b56-98f7-5a2eba02226f\") " Nov 04 01:38:54.385961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.385934 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca" pod "c938248d-ce29-4b56-98f7-5a2eba02226f" (UID: "c938248d-ce29-4b56-98f7-5a2eba02226f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:54.387785 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.387756 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg" pod "c938248d-ce29-4b56-98f7-5a2eba02226f" (UID: "c938248d-ce29-4b56-98f7-5a2eba02226f"). InnerVolumeSpecName "kube-api-access-nqbxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:54.387911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.387782 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca" pod "c938248d-ce29-4b56-98f7-5a2eba02226f" (UID: "c938248d-ce29-4b56-98f7-5a2eba02226f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:54.487457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.487377 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:54.487457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.487408 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c938248d-ce29-4b56-98f7-5a2eba02226f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:54.487457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:54.487420 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqbxg\" (UniqueName: \"kubernetes.io/projected/c938248d-ce29-4b56-98f7-5a2eba02226f-kube-api-access-nqbxg\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:55.213642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:55.213605 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" Nov 04 01:38:55.213844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:55.213638 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54" event={"ID":"c938248d-ce29-4b56-98f7-5a2eba02226f","Type":"ContainerDied","Data":"c8b174ccb5c6b956cce1343810f4157c0864b7f98865dd37ca85b58a272cee88"} Nov 04 01:38:55.213844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:38:55.213691 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b174ccb5c6b956cce1343810f4157c0864b7f98865dd37ca85b58a272cee88" Nov 04 01:39:13.837477 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:39:13.837419 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:39:21.105614 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:21.105563 4486 scope.go:122] "RemoveContainer" containerID="b0466ad0c26e05672d5e0b5b6ba64ea74d24f8b47e0df0ec001ae8eab954216a" Nov 04 01:39:21.115822 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:21.115800 4486 scope.go:122] "RemoveContainer" containerID="ef1d6ea6e9a75bf04af3e6164ce15a479b3d766ebc1739147d610aac084daa89" Nov 04 01:39:21.124783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:21.124757 4486 scope.go:122] "RemoveContainer" containerID="d1d503e0cc463791302f1b348ce283faee08a9062fca09417f0e3e817eb7c563" Nov 04 01:39:21.133718 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:21.133697 4486 scope.go:122] "RemoveContainer" containerID="7a513eb47eef572ebad7c446e31c3ade0b02820d3d6c3a4a59a59966977fc284" Nov 04 01:39:21.145347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:21.145322 4486 scope.go:122] "RemoveContainer" containerID="379b33f32e30ced139e73dc0ae1a29edb15cf45c001e6a918310f4fa15e2894d" Nov 04 01:39:22.540867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.540828 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb"] Nov 04 01:39:22.541399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.541345 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c938248d-ce29-4b56-98f7-5a2eba02226f" containerName="revoke-keys" Nov 04 01:39:22.541399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.541359 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c938248d-ce29-4b56-98f7-5a2eba02226f" containerName="revoke-keys" Nov 04 01:39:22.541480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.541421 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c938248d-ce29-4b56-98f7-5a2eba02226f" containerName="revoke-keys" Nov 04 01:39:22.547103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.547072 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.613386 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.613340 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.613583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.613509 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqj7\" (UniqueName: \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.613583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.613544 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.652358 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.652319 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:22.652542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.652378 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:39:22.655326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.655305 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb"] Nov 04 01:39:22.714366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.714329 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqj7\" (UniqueName: \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.714366 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.714368 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.714730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.714451 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.715119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.715097 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.716843 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.716824 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.769284 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.769254 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqj7\" (UniqueName: \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\") pod \"maas-api-revoke-sub-a4aef7d7-wsjpb\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:22.861460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:22.861422 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:23.235041 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:23.235009 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb"] Nov 04 01:39:23.236427 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:39:23.236390 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27df86f7_8441_4e0d_9f73_7fbbc3a28aaa.slice/crio-6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff WatchSource:0}: Error finding container 6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff: Status 404 returned error can't find the container with id 6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff Nov 04 01:39:23.403058 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:23.403017 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" event={"ID":"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa","Type":"ContainerStarted","Data":"6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff"} Nov 04 01:39:24.411586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:24.411544 4486 generic.go:356] "Generic (PLEG): container finished" podID="27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" containerID="41c0eae3f8d368188cdef317bb802d7cbeaa51c2d2e1349616c2e18831c5b9a1" exitCode=0 Nov 04 01:39:24.412023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:24.411602 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" event={"ID":"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa","Type":"ContainerDied","Data":"41c0eae3f8d368188cdef317bb802d7cbeaa51c2d2e1349616c2e18831c5b9a1"} Nov 04 01:39:25.560497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.560470 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:25.646197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.646154 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\") pod \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " Nov 04 01:39:25.646197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.646198 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\" (UniqueName: \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\") pod \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " Nov 04 01:39:25.646423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.646261 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\") pod \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\" (UID: \"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa\") " Nov 04 01:39:25.646710 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.646676 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca" pod "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" (UID: "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:25.648458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.648434 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca" pod "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" (UID: "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:25.648538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.648455 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7" pod "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" (UID: "27df86f7-8441-4e0d-9f73-7fbbc3a28aaa"). InnerVolumeSpecName "kube-api-access-nqqj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:25.746973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.746892 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.746973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.746923 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:25.746973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:25.746935 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqqj7\" (UniqueName: \"kubernetes.io/projected/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa-kube-api-access-nqqj7\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:26.424405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:26.424373 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" Nov 04 01:39:26.424604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:26.424372 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb" event={"ID":"27df86f7-8441-4e0d-9f73-7fbbc3a28aaa","Type":"ContainerDied","Data":"6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff"} Nov 04 01:39:26.424604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:26.424500 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce48b0b375bfbb7c93cdaa3db3b10fc5fb98b1e1563d1dbd2dad24c34c060ff" Nov 04 01:39:43.055719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:43.055676 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f"] Nov 04 01:39:43.103449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:43.103403 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6be64d0-m7j4f"] Nov 04 01:39:43.353672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:43.353619 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0" path="/var/lib/kubelet/pods/a1ad2415-abb7-4e2e-bdb2-fcb815fd39b0/volumes" Nov 04 01:39:43.908550 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:39:43.908474 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:39:56.312559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.312464 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp"] Nov 04 01:39:56.313222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.313033 4486 cpu_manager.go:414] "containerMap: removing container" podUID="27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" containerName="revoke-keys" Nov 04 01:39:56.313222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.313055 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" containerName="revoke-keys" Nov 04 01:39:56.313222 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.313142 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" containerName="revoke-keys" Nov 04 01:39:56.316844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.316824 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.366739 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.366693 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:39:56.366953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.366750 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:39:56.417138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.417094 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp"] Nov 04 01:39:56.424569 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.424541 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.424682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.424662 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82nv\" (UniqueName: \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.424748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.424735 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.525525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.525476 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.525748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.525544 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82nv\" (UniqueName: \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.525748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.525603 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.526294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.526262 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.527941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.527920 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.581014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.580890 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82nv\" (UniqueName: \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\") pod \"maas-api-revoke-sub-79805ca9-6fsmp\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:56.630839 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:56.630797 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:57.001619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:57.001589 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp"] Nov 04 01:39:57.003685 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:39:57.003659 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666d71b5_d4ab_4c77_9360_ccba4b86981d.slice/crio-58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d WatchSource:0}: Error finding container 58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d: Status 404 returned error can't find the container with id 58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d Nov 04 01:39:57.679550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:57.679437 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" event={"ID":"666d71b5-d4ab-4c77-9360-ccba4b86981d","Type":"ContainerStarted","Data":"11a114203287fd4c4f2eb6927f9ad89da98e2a65d42cb0ddcdf89802a296095e"} Nov 04 01:39:57.679550 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:57.679512 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" event={"ID":"666d71b5-d4ab-4c77-9360-ccba4b86981d","Type":"ContainerStarted","Data":"58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d"} Nov 04 01:39:57.791681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:57.791614 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" podStartSLOduration=1.791601 podStartE2EDuration="1.791601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:39:57.789868298 +0000 UTC m=+2566.548530461" watchObservedRunningTime="2026-11-04 01:39:57.791601 +0000 UTC m=+2566.550263161" Nov 04 01:39:58.687169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:58.687122 4486 generic.go:356] "Generic (PLEG): container finished" podID="666d71b5-d4ab-4c77-9360-ccba4b86981d" containerID="11a114203287fd4c4f2eb6927f9ad89da98e2a65d42cb0ddcdf89802a296095e" exitCode=0 Nov 04 01:39:58.687616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:58.687203 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" event={"ID":"666d71b5-d4ab-4c77-9360-ccba4b86981d","Type":"ContainerDied","Data":"11a114203287fd4c4f2eb6927f9ad89da98e2a65d42cb0ddcdf89802a296095e"} Nov 04 01:39:59.853820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.853793 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:39:59.961017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.960909 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\") pod \"666d71b5-d4ab-4c77-9360-ccba4b86981d\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " Nov 04 01:39:59.961194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.961087 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\") pod \"666d71b5-d4ab-4c77-9360-ccba4b86981d\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " Nov 04 01:39:59.961194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.961109 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\" (UniqueName: \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\") pod \"666d71b5-d4ab-4c77-9360-ccba4b86981d\" (UID: \"666d71b5-d4ab-4c77-9360-ccba4b86981d\") " Nov 04 01:39:59.961352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.961321 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca" pod "666d71b5-d4ab-4c77-9360-ccba4b86981d" (UID: "666d71b5-d4ab-4c77-9360-ccba4b86981d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:59.964079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.963975 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca" pod "666d71b5-d4ab-4c77-9360-ccba4b86981d" (UID: "666d71b5-d4ab-4c77-9360-ccba4b86981d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:59.964234 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:39:59.964091 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv" pod "666d71b5-d4ab-4c77-9360-ccba4b86981d" (UID: "666d71b5-d4ab-4c77-9360-ccba4b86981d"). InnerVolumeSpecName "kube-api-access-h82nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:00.062787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.062741 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.062787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.062783 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/666d71b5-d4ab-4c77-9360-ccba4b86981d-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.062787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.062793 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82nv\" (UniqueName: \"kubernetes.io/projected/666d71b5-d4ab-4c77-9360-ccba4b86981d-kube-api-access-h82nv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:00.701894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.701862 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" Nov 04 01:40:00.702095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.701894 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp" event={"ID":"666d71b5-d4ab-4c77-9360-ccba4b86981d","Type":"ContainerDied","Data":"58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d"} Nov 04 01:40:00.702095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:00.701950 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58dd7e4e7ad2154bedf894efab6c4bd2162e34cfdbfdc7050e6a2e690b181f4d" Nov 04 01:40:13.877692 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:40:13.877633 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:40:16.900244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.900206 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97"] Nov 04 01:40:16.900806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.900772 4486 cpu_manager.go:414] "containerMap: removing container" podUID="666d71b5-d4ab-4c77-9360-ccba4b86981d" containerName="revoke-keys" Nov 04 01:40:16.900806 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.900794 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666d71b5-d4ab-4c77-9360-ccba4b86981d" containerName="revoke-keys" Nov 04 01:40:16.901047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.900892 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="666d71b5-d4ab-4c77-9360-ccba4b86981d" containerName="revoke-keys" Nov 04 01:40:16.904290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.904268 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:16.958933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:16.958896 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97"] Nov 04 01:40:17.001243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.001208 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:17.001423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.001212 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:40:17.046143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.046098 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddtg\" (UniqueName: \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.046372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.046183 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.046372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.046362 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.147093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.147048 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddtg\" (UniqueName: \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.147312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.147135 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.147312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.147281 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.147846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.147816 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.149736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.149713 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.203670 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.203588 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddtg\" (UniqueName: \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\") pod \"maas-api-revoke-sub-0b7e8edb-fsm97\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.218339 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.218307 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:17.591286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.591251 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97"] Nov 04 01:40:17.592618 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:40:17.592591 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48d68c8_9f6f_4e5f_84e1_05a72d2eac44.slice/crio-90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3 WatchSource:0}: Error finding container 90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3: Status 404 returned error can't find the container with id 90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3 Nov 04 01:40:17.839008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:17.838939 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" event={"ID":"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44","Type":"ContainerStarted","Data":"90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3"} Nov 04 01:40:18.847035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:18.846995 4486 generic.go:356] "Generic (PLEG): container finished" podID="c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" containerID="4b7a98f3b256f5d68606ca37a905716bbd45f8bb70060e7809e745dc45df6b8c" exitCode=0 Nov 04 01:40:18.847035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:18.847030 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" event={"ID":"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44","Type":"ContainerDied","Data":"4b7a98f3b256f5d68606ca37a905716bbd45f8bb70060e7809e745dc45df6b8c"} Nov 04 01:40:19.997380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:19.997353 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:20.077748 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.077708 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\") pod \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " Nov 04 01:40:20.077926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.077802 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\") pod \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " Nov 04 01:40:20.077926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.077861 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\" (UniqueName: \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\") pod \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\" (UID: \"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44\") " Nov 04 01:40:20.078162 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.078136 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca" pod "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" (UID: "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:20.080017 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.079971 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca" pod "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" (UID: "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:20.080143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.080125 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg" pod "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" (UID: "c48d68c8-9f6f-4e5f-84e1-05a72d2eac44"). InnerVolumeSpecName "kube-api-access-nddtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:20.179059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.178945 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:20.179059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.179001 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddtg\" (UniqueName: \"kubernetes.io/projected/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-kube-api-access-nddtg\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:20.179059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.179012 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:20.860599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.860569 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" Nov 04 01:40:20.860762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.860604 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97" event={"ID":"c48d68c8-9f6f-4e5f-84e1-05a72d2eac44","Type":"ContainerDied","Data":"90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3"} Nov 04 01:40:20.860762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:20.860652 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f00b4c106eeaee4f5a32e2460e1dc9ea080711d4790124ecd1c66616392ca3" Nov 04 01:40:21.193907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:21.193826 4486 scope.go:122] "RemoveContainer" containerID="144b97d0a66af5adab7026f76f6d3bf16871d3ae87083262654a531ba35b6f7b" Nov 04 01:40:21.203777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:21.203755 4486 scope.go:122] "RemoveContainer" containerID="82769f51317645d859b0afbb603a63bc4f20f5a5e83594e2631647e50d851d71" Nov 04 01:40:21.212780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:21.212761 4486 scope.go:122] "RemoveContainer" containerID="e5322b0121ebaeb84a8f85856da8a12de3a2053b14f2031190b78ff7e27c5153" Nov 04 01:40:28.228065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.228018 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr"] Nov 04 01:40:28.228619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.228522 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" containerName="revoke-keys" Nov 04 01:40:28.228619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.228537 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" containerName="revoke-keys" Nov 04 01:40:28.228714 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.228639 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" containerName="revoke-keys" Nov 04 01:40:28.233772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.233750 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.287666 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.287632 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr"] Nov 04 01:40:28.331105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.331054 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:28.331105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.331061 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:40:28.361816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.361779 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.362061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.361909 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.362061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.361947 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcssr\" (UniqueName: \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.463484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.463441 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.463700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.463494 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcssr\" (UniqueName: \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.463700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.463617 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.464231 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.464206 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.466015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.465978 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.518344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.518250 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcssr\" (UniqueName: \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\") pod \"maas-api-revoke-sub-3d4156f1-xngtr\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.547911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.547863 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:28.916622 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:28.916591 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr"] Nov 04 01:40:28.918154 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:40:28.918124 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a0af92_2ac4_4907_84aa_b79ede243dd6.slice/crio-d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede WatchSource:0}: Error finding container d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede: Status 404 returned error can't find the container with id d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede Nov 04 01:40:29.919606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:29.919560 4486 generic.go:356] "Generic (PLEG): container finished" podID="54a0af92-2ac4-4907-84aa-b79ede243dd6" containerID="91ff0dfd9ea0c5f0ad4f6f1d61781cb4e7193d66160a5957f59cf0b251897ad1" exitCode=0 Nov 04 01:40:29.920101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:29.919637 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" event={"ID":"54a0af92-2ac4-4907-84aa-b79ede243dd6","Type":"ContainerDied","Data":"91ff0dfd9ea0c5f0ad4f6f1d61781cb4e7193d66160a5957f59cf0b251897ad1"} Nov 04 01:40:29.920101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:29.919698 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" event={"ID":"54a0af92-2ac4-4907-84aa-b79ede243dd6","Type":"ContainerStarted","Data":"d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede"} Nov 04 01:40:31.509010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.508102 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:31.543007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.540395 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\" (UniqueName: \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\") pod \"54a0af92-2ac4-4907-84aa-b79ede243dd6\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " Nov 04 01:40:31.543007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.541792 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\") pod \"54a0af92-2ac4-4907-84aa-b79ede243dd6\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " Nov 04 01:40:31.543007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.541858 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\") pod \"54a0af92-2ac4-4907-84aa-b79ede243dd6\" (UID: \"54a0af92-2ac4-4907-84aa-b79ede243dd6\") " Nov 04 01:40:31.543007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.542742 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca" pod "54a0af92-2ac4-4907-84aa-b79ede243dd6" (UID: "54a0af92-2ac4-4907-84aa-b79ede243dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:31.603005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.599033 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr" pod "54a0af92-2ac4-4907-84aa-b79ede243dd6" (UID: "54a0af92-2ac4-4907-84aa-b79ede243dd6"). InnerVolumeSpecName "kube-api-access-bcssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:31.603005 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.601946 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca" pod "54a0af92-2ac4-4907-84aa-b79ede243dd6" (UID: "54a0af92-2ac4-4907-84aa-b79ede243dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:31.647008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.643174 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcssr\" (UniqueName: \"kubernetes.io/projected/54a0af92-2ac4-4907-84aa-b79ede243dd6-kube-api-access-bcssr\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.647008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.643218 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.647008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.643235 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a0af92-2ac4-4907-84aa-b79ede243dd6-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:31.961874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.961830 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" event={"ID":"54a0af92-2ac4-4907-84aa-b79ede243dd6","Type":"ContainerDied","Data":"d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede"} Nov 04 01:40:31.970103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.970062 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46884f0bf193fcc17d05c1072ad5b879830a3046aa5f1fd571e4892fbd86ede" Nov 04 01:40:31.970443 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:31.970420 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr" Nov 04 01:40:43.841184 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:40:43.841135 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:40:49.351831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.351797 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr"] Nov 04 01:40:49.398890 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.398842 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xngtr"] Nov 04 01:40:49.435470 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.435433 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt"] Nov 04 01:40:49.435916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.435897 4486 cpu_manager.go:414] "containerMap: removing container" podUID="54a0af92-2ac4-4907-84aa-b79ede243dd6" containerName="revoke-keys" Nov 04 01:40:49.435958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.435918 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a0af92-2ac4-4907-84aa-b79ede243dd6" containerName="revoke-keys" Nov 04 01:40:49.436025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.435993 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a0af92-2ac4-4907-84aa-b79ede243dd6" containerName="revoke-keys" Nov 04 01:40:49.439402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.439378 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.479045 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.479001 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.479235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.479055 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglrf\" (UniqueName: \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.479281 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.479222 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.490639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.490606 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:40:49.490639 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.490609 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:40:49.493800 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.493773 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt"] Nov 04 01:40:49.580325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.580283 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.580553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.580377 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.580553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.580405 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglrf\" (UniqueName: \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.581029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.580968 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.582852 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.582828 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.635184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.635111 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglrf\" (UniqueName: \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\") pod \"maas-api-revoke-sub-3d4156f1-vxtdt\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:49.752510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:49.752474 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:50.128327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:50.128287 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt"] Nov 04 01:40:50.128851 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:40:50.128825 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff4bcf1_cbbf_41b8_ae53_cac2f0fd74b8.slice/crio-644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b WatchSource:0}: Error finding container 644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b: Status 404 returned error can't find the container with id 644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b Nov 04 01:40:51.117079 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:51.117008 4486 generic.go:356] "Generic (PLEG): container finished" podID="bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" containerID="023c629758bf5ae4078f11fb1397ed3502354a9ed668bce469e9ebdcfc30abda" exitCode=0 Nov 04 01:40:51.117497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:51.117081 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" event={"ID":"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8","Type":"ContainerDied","Data":"023c629758bf5ae4078f11fb1397ed3502354a9ed668bce469e9ebdcfc30abda"} Nov 04 01:40:51.117497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:51.117139 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" event={"ID":"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8","Type":"ContainerStarted","Data":"644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b"} Nov 04 01:40:51.355401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:51.355354 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a0af92-2ac4-4907-84aa-b79ede243dd6" path="/var/lib/kubelet/pods/54a0af92-2ac4-4907-84aa-b79ede243dd6/volumes" Nov 04 01:40:52.262829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.262796 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:52.309603 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.309565 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\") pod \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " Nov 04 01:40:52.309793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.309617 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\") pod \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " Nov 04 01:40:52.309793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.309638 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\" (UniqueName: \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\") pod \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\" (UID: \"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8\") " Nov 04 01:40:52.310026 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.309997 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca" pod "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" (UID: "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:52.311730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.311696 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca" pod "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" (UID: "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:52.311835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.311732 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf" pod "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" (UID: "bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8"). InnerVolumeSpecName "kube-api-access-mglrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:52.411423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.411323 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:52.411423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.411359 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:52.411423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:52.411374 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mglrf\" (UniqueName: \"kubernetes.io/projected/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8-kube-api-access-mglrf\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.130934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:53.130898 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" Nov 04 01:40:53.131115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:53.130934 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt" event={"ID":"bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8","Type":"ContainerDied","Data":"644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b"} Nov 04 01:40:53.131115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:40:53.131012 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644ddd94ddc06f9685d305d895b5e6b220601461f6738a90e8de29de04692e8b" Nov 04 01:41:13.831568 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:41:13.831527 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:41:21.243912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:21.243870 4486 scope.go:122] "RemoveContainer" containerID="2190e660987deb48f6abdaee215aeee8e4842976fdbad6a99810ba5fb11b5f46" Nov 04 01:41:27.141329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.141223 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln"] Nov 04 01:41:27.141886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.141865 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" containerName="revoke-keys" Nov 04 01:41:27.141886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.141885 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" containerName="revoke-keys" Nov 04 01:41:27.142057 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.141993 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" containerName="revoke-keys" Nov 04 01:41:27.146798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.146768 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.186523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.186486 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg97v\" (UniqueName: \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.186698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.186601 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.186698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.186625 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.197836 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.197802 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln"] Nov 04 01:41:27.240777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.240737 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:41:27.240960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.240801 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:41:27.288322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.288289 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg97v\" (UniqueName: \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.288523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.288358 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.288523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.288377 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.289090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.289055 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.290828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.290808 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.343427 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.343386 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg97v\" (UniqueName: \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\") pod \"maas-api-revoke-sub-e2fc9c5b-9sjln\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.462033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.461923 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:27.833528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:27.833493 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln"] Nov 04 01:41:27.835043 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:41:27.835008 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3861eebd_ba5d_4626_be76_2c09b4ddd78f.slice/crio-28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc WatchSource:0}: Error finding container 28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc: Status 404 returned error can't find the container with id 28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc Nov 04 01:41:28.363137 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:28.363094 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" event={"ID":"3861eebd-ba5d-4626-be76-2c09b4ddd78f","Type":"ContainerStarted","Data":"2ed3af2836fc327c1665f5fab17d84c1feda766b4c56b005ce3969f8930e4e5f"} Nov 04 01:41:28.363528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:28.363155 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" event={"ID":"3861eebd-ba5d-4626-be76-2c09b4ddd78f","Type":"ContainerStarted","Data":"28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc"} Nov 04 01:41:28.481074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:28.480937 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" podStartSLOduration=1.480921998 podStartE2EDuration="1.480921998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:28.478907476 +0000 UTC m=+2657.237569632" watchObservedRunningTime="2026-11-04 01:41:28.480921998 +0000 UTC m=+2657.239584155" Nov 04 01:41:29.374158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:29.374124 4486 generic.go:356] "Generic (PLEG): container finished" podID="3861eebd-ba5d-4626-be76-2c09b4ddd78f" containerID="2ed3af2836fc327c1665f5fab17d84c1feda766b4c56b005ce3969f8930e4e5f" exitCode=0 Nov 04 01:41:29.374581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:29.374217 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" event={"ID":"3861eebd-ba5d-4626-be76-2c09b4ddd78f","Type":"ContainerDied","Data":"2ed3af2836fc327c1665f5fab17d84c1feda766b4c56b005ce3969f8930e4e5f"} Nov 04 01:41:30.523518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.523482 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:30.624957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.624916 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\" (UniqueName: \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\") pod \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " Nov 04 01:41:30.625197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.625022 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\") pod \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " Nov 04 01:41:30.625197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.625046 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\") pod \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\" (UID: \"3861eebd-ba5d-4626-be76-2c09b4ddd78f\") " Nov 04 01:41:30.625471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.625440 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca" pod "3861eebd-ba5d-4626-be76-2c09b4ddd78f" (UID: "3861eebd-ba5d-4626-be76-2c09b4ddd78f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:30.627270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.627230 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v" pod "3861eebd-ba5d-4626-be76-2c09b4ddd78f" (UID: "3861eebd-ba5d-4626-be76-2c09b4ddd78f"). InnerVolumeSpecName "kube-api-access-rg97v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:30.627403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.627330 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca" pod "3861eebd-ba5d-4626-be76-2c09b4ddd78f" (UID: "3861eebd-ba5d-4626-be76-2c09b4ddd78f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:30.725916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.725874 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:30.725916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.725909 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3861eebd-ba5d-4626-be76-2c09b4ddd78f-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:30.725916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:30.725919 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg97v\" (UniqueName: \"kubernetes.io/projected/3861eebd-ba5d-4626-be76-2c09b4ddd78f-kube-api-access-rg97v\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:31.387898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:31.387862 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" Nov 04 01:41:31.387898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:31.387889 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln" event={"ID":"3861eebd-ba5d-4626-be76-2c09b4ddd78f","Type":"ContainerDied","Data":"28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc"} Nov 04 01:41:31.388140 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:31.387942 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28c2b0ead5e24639edaeded4c9c667da0ea0bc19d69213edec57d6da5f9858fc" Nov 04 01:41:37.904046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.904008 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f"] Nov 04 01:41:37.905061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.904732 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3861eebd-ba5d-4626-be76-2c09b4ddd78f" containerName="revoke-keys" Nov 04 01:41:37.905061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.904755 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3861eebd-ba5d-4626-be76-2c09b4ddd78f" containerName="revoke-keys" Nov 04 01:41:37.905061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.904861 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3861eebd-ba5d-4626-be76-2c09b4ddd78f" containerName="revoke-keys" Nov 04 01:41:37.908260 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.908238 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:37.962875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.962838 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f"] Nov 04 01:41:37.996299 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.996262 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:37.996492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.996304 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vv9t\" (UniqueName: \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:37.996492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:37.996442 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.005434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.005405 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:41:38.005526 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.005485 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:41:38.097147 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.097109 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.097330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.097276 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.097330 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.097303 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vv9t\" (UniqueName: \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.097904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.097883 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.099687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.099658 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.153105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.153069 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vv9t\" (UniqueName: \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\") pod \"maas-api-revoke-sub-5a29efb3-tjw5f\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.222605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.222503 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:38.594453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:38.594420 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f"] Nov 04 01:41:38.596100 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:41:38.596063 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b55bc3_7d0b_467b_9b50_c6b5bb8c50f5.slice/crio-9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479 WatchSource:0}: Error finding container 9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479: Status 404 returned error can't find the container with id 9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479 Nov 04 01:41:39.438106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:39.438068 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" event={"ID":"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5","Type":"ContainerStarted","Data":"5e47e392b646180ba1f41677027f8ed0c29f203ae234991e68fdf8d6b77cbd9c"} Nov 04 01:41:39.438537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:39.438123 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" event={"ID":"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5","Type":"ContainerStarted","Data":"9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479"} Nov 04 01:41:40.446377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:40.446338 4486 generic.go:356] "Generic (PLEG): container finished" podID="36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" containerID="5e47e392b646180ba1f41677027f8ed0c29f203ae234991e68fdf8d6b77cbd9c" exitCode=0 Nov 04 01:41:40.446875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:40.446421 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" event={"ID":"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5","Type":"ContainerDied","Data":"5e47e392b646180ba1f41677027f8ed0c29f203ae234991e68fdf8d6b77cbd9c"} Nov 04 01:41:41.833367 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:41.833334 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:41.982691 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:41.982641 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\" (UniqueName: \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\") pod \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " Nov 04 01:41:41.983092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:41.983052 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\") pod \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " Nov 04 01:41:41.983344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:41.983324 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\") pod \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\" (UID: \"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5\") " Nov 04 01:41:41.986874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:41.986846 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca" pod "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" (UID: "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:42.015866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.015819 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t" pod "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" (UID: "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5"). InnerVolumeSpecName "kube-api-access-4vv9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:42.018707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.018667 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca" pod "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" (UID: "36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:42.107011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.087686 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:42.107011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.087731 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:42.107011 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.087747 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vv9t\" (UniqueName: \"kubernetes.io/projected/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5-kube-api-access-4vv9t\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:42.481075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.480586 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" event={"ID":"36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5","Type":"ContainerDied","Data":"9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479"} Nov 04 01:41:42.481075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.480659 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4760af2818d3d93ea919d0a3c78104bd8fc4a1b79807f6f28d90c8a61fd479" Nov 04 01:41:42.481075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:42.480730 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f" Nov 04 01:41:43.831461 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:41:43.831418 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:41:50.046637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:50.046573 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2"] Nov 04 01:41:50.094098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:50.094060 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-454k2"] Nov 04 01:41:51.347780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:41:51.347745 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d4b781-b199-41a6-9f2b-90261b175140" path="/var/lib/kubelet/pods/64d4b781-b199-41a6-9f2b-90261b175140/volumes" Nov 04 01:42:01.100769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.100734 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt"] Nov 04 01:42:01.147995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.147947 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vxtdt"] Nov 04 01:42:01.348308 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.348268 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8" path="/var/lib/kubelet/pods/bff4bcf1-cbbf-41b8-ae53-cac2f0fd74b8/volumes" Nov 04 01:42:01.460387 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.460294 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2"] Nov 04 01:42:01.460766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.460745 4486 cpu_manager.go:414] "containerMap: removing container" podUID="36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" containerName="revoke-keys" Nov 04 01:42:01.460766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.460759 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" containerName="revoke-keys" Nov 04 01:42:01.460863 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.460841 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" containerName="revoke-keys" Nov 04 01:42:01.465454 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.465433 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.515506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.515469 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:42:01.562493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.562457 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:42:01.565507 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.565477 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2"] Nov 04 01:42:01.597921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.597892 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.598138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.597947 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlt72\" (UniqueName: \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.598138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.598122 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.699481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.699448 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.699680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.699504 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlt72\" (UniqueName: \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.699680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.699550 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.700216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.700191 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.701907 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.701884 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.753676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.753607 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlt72\" (UniqueName: \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\") pod \"maas-api-revoke-sub-3d4156f1-fn9j2\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:01.779434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:01.779402 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:02.155471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:02.155426 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2"] Nov 04 01:42:02.156812 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:42:02.156784 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa995a60_1848_4b21_9709_bfa6ebae2dd0.slice/crio-1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac WatchSource:0}: Error finding container 1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac: Status 404 returned error can't find the container with id 1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac Nov 04 01:42:02.606901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:02.606860 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" event={"ID":"fa995a60-1848-4b21-9709-bfa6ebae2dd0","Type":"ContainerStarted","Data":"504cc52db8d34e19d05f0faa205b29e58a59accba2d878555ec3729d299e5dec"} Nov 04 01:42:02.607099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:02.606929 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" event={"ID":"fa995a60-1848-4b21-9709-bfa6ebae2dd0","Type":"ContainerStarted","Data":"1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac"} Nov 04 01:42:02.718934 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:02.718812 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" podStartSLOduration=1.718799073 podStartE2EDuration="1.718799073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:02.71692981 +0000 UTC m=+2691.475591973" watchObservedRunningTime="2026-11-04 01:42:02.718799073 +0000 UTC m=+2691.477461233" Nov 04 01:42:03.614638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:03.614600 4486 generic.go:356] "Generic (PLEG): container finished" podID="fa995a60-1848-4b21-9709-bfa6ebae2dd0" containerID="504cc52db8d34e19d05f0faa205b29e58a59accba2d878555ec3729d299e5dec" exitCode=0 Nov 04 01:42:03.615135 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:03.614684 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" event={"ID":"fa995a60-1848-4b21-9709-bfa6ebae2dd0","Type":"ContainerDied","Data":"504cc52db8d34e19d05f0faa205b29e58a59accba2d878555ec3729d299e5dec"} Nov 04 01:42:04.765777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.765745 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:04.830445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.830406 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\") pod \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " Nov 04 01:42:04.830629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.830506 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\") pod \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " Nov 04 01:42:04.830629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.830569 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\" (UniqueName: \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\") pod \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\" (UID: \"fa995a60-1848-4b21-9709-bfa6ebae2dd0\") " Nov 04 01:42:04.830943 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.830920 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca" pod "fa995a60-1848-4b21-9709-bfa6ebae2dd0" (UID: "fa995a60-1848-4b21-9709-bfa6ebae2dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:04.832510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.832482 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca" pod "fa995a60-1848-4b21-9709-bfa6ebae2dd0" (UID: "fa995a60-1848-4b21-9709-bfa6ebae2dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:42:04.832644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.832622 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72" pod "fa995a60-1848-4b21-9709-bfa6ebae2dd0" (UID: "fa995a60-1848-4b21-9709-bfa6ebae2dd0"). InnerVolumeSpecName "kube-api-access-qlt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:04.932158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.932077 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:04.932158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.932107 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa995a60-1848-4b21-9709-bfa6ebae2dd0-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:04.932158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:04.932123 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlt72\" (UniqueName: \"kubernetes.io/projected/fa995a60-1848-4b21-9709-bfa6ebae2dd0-kube-api-access-qlt72\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:05.628233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:05.628134 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" Nov 04 01:42:05.628233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:05.628171 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2" event={"ID":"fa995a60-1848-4b21-9709-bfa6ebae2dd0","Type":"ContainerDied","Data":"1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac"} Nov 04 01:42:05.628233 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:05.628223 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1411c312af8baef520092d0c63a1a0801a739594e30637a37e063f3de65761ac" Nov 04 01:42:13.830430 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:42:13.830391 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:42:18.048108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.048080 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:42:18.063173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.063145 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:42:18.079351 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.079317 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:42:18.096291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.096263 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:42:18.121444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.121403 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:42:18.121927 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.121907 4486 cpu_manager.go:414] "containerMap: removing container" podUID="fa995a60-1848-4b21-9709-bfa6ebae2dd0" containerName="revoke-keys" Nov 04 01:42:18.122060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.121928 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa995a60-1848-4b21-9709-bfa6ebae2dd0" containerName="revoke-keys" Nov 04 01:42:18.122125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.122104 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa995a60-1848-4b21-9709-bfa6ebae2dd0" containerName="revoke-keys" Nov 04 01:42:18.124961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.124938 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.176289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.176235 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f9348a-openshift-default-dockercfg-nkckz\"" Nov 04 01:42:18.227336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.227294 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:42:18.263318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263280 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263325 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263358 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263479 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263521 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263558 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmpx\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263641 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263780 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263744 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.263828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.263814 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.364720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364680 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.364720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364723 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364747 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364782 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmpx\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364822 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364885 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.364968 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365034 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365064 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365271 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365308 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365438 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.365947 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.365926 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.366150 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.366124 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.367407 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.367377 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.367644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.367625 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.423250 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.423217 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.423673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.423649 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmpx\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\") pod \"e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.446400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.446362 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:18.867012 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:42:18.866940 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dd3754_5fae_4d7c_ab93_349f83bf4017.slice/crio-f4bd82303030661a3c11b8947aef89906c74c764c61c151f1aea7798f3f66cc0 WatchSource:0}: Error finding container f4bd82303030661a3c11b8947aef89906c74c764c61c151f1aea7798f3f66cc0: Status 404 returned error can't find the container with id f4bd82303030661a3c11b8947aef89906c74c764c61c151f1aea7798f3f66cc0 Nov 04 01:42:18.867627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.867603 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:42:18.869301 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.869265 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:18.869396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.869335 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:18.869396 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:18.869364 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:19.718163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:19.718125 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerStarted","Data":"303bc63cd5b41fa9ff181f0de86ddee79a8d3c6b7f6f8010f3e28fa079596080"} Nov 04 01:42:19.718616 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:19.718181 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerStarted","Data":"f4bd82303030661a3c11b8947aef89906c74c764c61c151f1aea7798f3f66cc0"} Nov 04 01:42:19.925672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:19.925613 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podStartSLOduration=1.925598801 podStartE2EDuration="1.925598801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:19.875869819 +0000 UTC m=+2708.634531982" watchObservedRunningTime="2026-11-04 01:42:19.925598801 +0000 UTC m=+2708.684261020" Nov 04 01:42:20.447615 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:20.447574 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:20.453356 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:20.453329 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:20.725201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:20.725118 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:20.726228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:20.726203 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:21.050046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:21.049935 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f"] Nov 04 01:42:21.052964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:21.052936 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5n26f"] Nov 04 01:42:21.349391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:21.349355 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7" path="/var/lib/kubelet/pods/e1ad6e08-b951-44f6-a7a3-6e8b53e5fdd7/volumes" Nov 04 01:42:33.045002 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:33.044852 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8"] Nov 04 01:42:33.091887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:33.091842 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-vsvn8"] Nov 04 01:42:33.347675 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:33.347630 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5692ffc-8e81-4f80-a861-e05d46f3c1a4" path="/var/lib/kubelet/pods/b5692ffc-8e81-4f80-a861-e05d46f3c1a4/volumes" Nov 04 01:42:43.832839 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:42:43.832802 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:42:46.914759 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.914726 4486 generic.go:356] "Generic (PLEG): container finished" podID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerID="303bc63cd5b41fa9ff181f0de86ddee79a8d3c6b7f6f8010f3e28fa079596080" exitCode=0 Nov 04 01:42:46.915226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.914798 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerDied","Data":"303bc63cd5b41fa9ff181f0de86ddee79a8d3c6b7f6f8010f3e28fa079596080"} Nov 04 01:42:46.915226 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.915195 4486 scope.go:122] "RemoveContainer" containerID="303bc63cd5b41fa9ff181f0de86ddee79a8d3c6b7f6f8010f3e28fa079596080" Nov 04 01:42:46.916503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.916477 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:46.916561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.916549 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:46.916602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:46.916585 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:42:47.923719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:47.923684 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerStarted","Data":"5213494f88323ab5a7804f40b029f4138aed745b99760f7b466895734b253592"} Nov 04 01:42:48.453431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:48.453261 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:48.454611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:48.454574 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 04 01:42:48.454777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:48.454637 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 04 01:42:49.452009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:49.451468 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 04 01:42:49.452009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:49.451552 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 04 01:42:50.450464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:50.450422 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 04 01:42:50.450799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:50.450764 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 04 01:42:50.725818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:50.725726 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:51.450009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.449491 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 04 01:42:51.450009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.449561 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 04 01:42:51.685841 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.685798 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:42:51.689194 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.689168 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.762620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.762521 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:42:51.797163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.797116 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f9348a\"/\"openshift-service-ca.crt\"" Nov 04 01:42:51.797163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.797154 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f9348a\"/\"kube-root-ca.crt\"" Nov 04 01:42:51.797163 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.797152 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f9348a\"/\"simulator-single-main-f9348a-kserve-self-signed-certs\"" Nov 04 01:42:51.797437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.797225 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f9348a\"/\"default-dockercfg-qbpfw\"" Nov 04 01:42:51.881146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881069 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.881146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881153 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.881418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881194 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.881418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881236 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.881418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881275 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkx7f\" (UniqueName: \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.881418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.881293 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982196 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982303 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982355 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkx7f\" (UniqueName: \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982386 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982485 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982462 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982502 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982726 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.982877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.982854 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.983065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.983038 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.984799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.984773 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:51.987066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:51.987044 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:52.056267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.056182 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkx7f\" (UniqueName: \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\") pod \"simulator-single-main-f9348a-kserve-7c66794755-l8m2t\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:52.300565 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.300523 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:52.453071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.452489 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Nov 04 01:42:52.453071 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.452554 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Nov 04 01:42:52.694430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.694395 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:42:52.695839 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:42:52.695807 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79f403d7_0ed7_4ce9_aca6_8f3ab0da8dfe.slice/crio-91a42b7d0fb95c7e588232e6da17ec2029ba96810859a0a81341c53efff60ba7 WatchSource:0}: Error finding container 91a42b7d0fb95c7e588232e6da17ec2029ba96810859a0a81341c53efff60ba7: Status 404 returned error can't find the container with id 91a42b7d0fb95c7e588232e6da17ec2029ba96810859a0a81341c53efff60ba7 Nov 04 01:42:52.996003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.995849 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" event={"ID":"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe","Type":"ContainerStarted","Data":"be9bde80772cab2f0d37a598618646625df201ee313699899535e8a45e3f5185"} Nov 04 01:42:52.996003 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:52.995901 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" event={"ID":"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe","Type":"ContainerStarted","Data":"91a42b7d0fb95c7e588232e6da17ec2029ba96810859a0a81341c53efff60ba7"} Nov 04 01:42:53.114795 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:53.114728 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" podStartSLOduration=2.114712477 podStartE2EDuration="2.114712477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:53.112315893 +0000 UTC m=+2741.870978056" watchObservedRunningTime="2026-11-04 01:42:53.114712477 +0000 UTC m=+2741.873374637" Nov 04 01:42:53.301506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:53.301418 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:42:53.451838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:53.451805 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:42:53.452829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:42:53.452806 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:43:02.301306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:02.301255 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:43:02.314323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:02.314294 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:43:03.077722 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:03.077672 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:43:09.045716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:09.045674 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn"] Nov 04 01:43:09.093064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:09.093030 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nzvgn"] Nov 04 01:43:09.354995 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:09.349827 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc" path="/var/lib/kubelet/pods/7f147fd6-d73d-4b0e-a1fb-b7eced7cf6fc/volumes" Nov 04 01:43:13.830780 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:43:13.830738 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:43:20.045742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:20.045702 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw"] Nov 04 01:43:20.092955 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:20.092915 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-sm8qw"] Nov 04 01:43:21.278712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:21.278673 4486 scope.go:122] "RemoveContainer" containerID="4b4ff93872954b1d57f65e2345f9c2053b50e29f6399f9b7b0ac82eb11b4f37c" Nov 04 01:43:21.289388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:21.289360 4486 scope.go:122] "RemoveContainer" containerID="809fca64729bc1fe2e81ddaf48f86f427aa595e698f31382a868977e48952285" Nov 04 01:43:21.347684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:21.347651 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c17621-85f9-4246-860f-5b9c15d604e8" path="/var/lib/kubelet/pods/e9c17621-85f9-4246-860f-5b9c15d604e8/volumes" Nov 04 01:43:40.659619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.659579 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh"] Nov 04 01:43:40.663347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.663324 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.704853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.704813 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b9l\" (UniqueName: \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.705053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.704871 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.705053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.704977 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.713169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.713143 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:43:40.760520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.760481 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:43:40.764267 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.764234 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh"] Nov 04 01:43:40.806189 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.806150 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b9l\" (UniqueName: \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.806381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.806202 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.806381 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.806241 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.806931 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.806898 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.808753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.808729 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.861477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.861441 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b9l\" (UniqueName: \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\") pod \"maas-api-revoke-sub-2811e627-znfmh\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:40.977346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:40.977254 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:41.348818 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:43:41.348785 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142a5a5a_0a63_4e06_aec9_8cb86b8a5e06.slice/crio-914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54 WatchSource:0}: Error finding container 914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54: Status 404 returned error can't find the container with id 914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54 Nov 04 01:43:41.349900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:41.349875 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh"] Nov 04 01:43:42.321181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:42.320970 4486 generic.go:356] "Generic (PLEG): container finished" podID="142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" containerID="65415ebab2ad1545f4d612fa64e609701b9935a77e07b6de6102ee7778486a13" exitCode=0 Nov 04 01:43:42.321181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:42.321027 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" event={"ID":"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06","Type":"ContainerDied","Data":"65415ebab2ad1545f4d612fa64e609701b9935a77e07b6de6102ee7778486a13"} Nov 04 01:43:42.321181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:42.321083 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" event={"ID":"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06","Type":"ContainerStarted","Data":"914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54"} Nov 04 01:43:43.472589 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.472562 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:43.535031 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.534971 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\") pod \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " Nov 04 01:43:43.535220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.535045 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\") pod \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " Nov 04 01:43:43.535220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.535104 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\" (UniqueName: \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\") pod \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\" (UID: \"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06\") " Nov 04 01:43:43.535438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.535406 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca" pod "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" (UID: "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:43.537221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.537196 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca" pod "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" (UID: "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:43.537335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.537272 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l" pod "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" (UID: "142a5a5a-0a63-4e06-aec9-8cb86b8a5e06"). InnerVolumeSpecName "kube-api-access-99b9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:43.636816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.636729 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b9l\" (UniqueName: \"kubernetes.io/projected/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-kube-api-access-99b9l\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:43.636816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.636762 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:43.636816 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:43.636809 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:43.830951 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:43:43.830909 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:43:44.336033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:44.335999 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" Nov 04 01:43:44.336033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:44.336018 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh" event={"ID":"142a5a5a-0a63-4e06-aec9-8cb86b8a5e06","Type":"ContainerDied","Data":"914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54"} Nov 04 01:43:44.336239 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:44.336076 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914c7552ccef5ccd440933af41bc2f963d658b902c79a83115a10104fa2acf54" Nov 04 01:43:52.861244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.861204 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb"] Nov 04 01:43:52.861779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.861687 4486 cpu_manager.go:414] "containerMap: removing container" podUID="142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" containerName="revoke-keys" Nov 04 01:43:52.861779 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.861701 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" containerName="revoke-keys" Nov 04 01:43:52.861859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.861795 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" containerName="revoke-keys" Nov 04 01:43:52.865382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.865364 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:52.918395 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.918356 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb"] Nov 04 01:43:52.961449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.961415 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:43:52.961449 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:52.961431 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:43:53.039474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.039437 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.039662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.039523 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmlz\" (UniqueName: \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.039662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.039588 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.141360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.141272 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.141523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.141357 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmlz\" (UniqueName: \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.141523 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.141407 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.142060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.142037 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.143708 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.143682 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.196316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.196284 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmlz\" (UniqueName: \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.479169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.479079 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:53.852470 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:43:53.852436 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda158d236_9b43_4c8a_9203_74f1a3781b76.slice/crio-059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949 WatchSource:0}: Error finding container 059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949: Status 404 returned error can't find the container with id 059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949 Nov 04 01:43:53.852625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:53.852471 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb"] Nov 04 01:43:54.401786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:54.401684 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" event={"ID":"a158d236-9b43-4c8a-9203-74f1a3781b76","Type":"ContainerStarted","Data":"2b9ab38219443664083b2114034119b776d8248ecbf1428839d4ca775aadde7f"} Nov 04 01:43:54.401786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:54.401744 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" event={"ID":"a158d236-9b43-4c8a-9203-74f1a3781b76","Type":"ContainerStarted","Data":"059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949"} Nov 04 01:43:54.513467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:54.513409 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" podStartSLOduration=2.513395536 podStartE2EDuration="2.513395536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:43:54.509770373 +0000 UTC m=+2803.268432525" watchObservedRunningTime="2026-11-04 01:43:54.513395536 +0000 UTC m=+2803.272057698" Nov 04 01:43:55.057559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:55.057406 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54"] Nov 04 01:43:55.059651 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:55.059614 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mdm54"] Nov 04 01:43:55.349004 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:55.348943 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c938248d-ce29-4b56-98f7-5a2eba02226f" path="/var/lib/kubelet/pods/c938248d-ce29-4b56-98f7-5a2eba02226f/volumes" Nov 04 01:43:55.409715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:55.409677 4486 generic.go:356] "Generic (PLEG): container finished" podID="a158d236-9b43-4c8a-9203-74f1a3781b76" containerID="2b9ab38219443664083b2114034119b776d8248ecbf1428839d4ca775aadde7f" exitCode=0 Nov 04 01:43:55.410129 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:55.409714 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" event={"ID":"a158d236-9b43-4c8a-9203-74f1a3781b76","Type":"ContainerDied","Data":"2b9ab38219443664083b2114034119b776d8248ecbf1428839d4ca775aadde7f"} Nov 04 01:43:56.566103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.566071 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:56.677409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.677326 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\" (UniqueName: \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\") pod \"a158d236-9b43-4c8a-9203-74f1a3781b76\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " Nov 04 01:43:56.677627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.677419 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\") pod \"a158d236-9b43-4c8a-9203-74f1a3781b76\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " Nov 04 01:43:56.677627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.677560 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\") pod \"a158d236-9b43-4c8a-9203-74f1a3781b76\" (UID: \"a158d236-9b43-4c8a-9203-74f1a3781b76\") " Nov 04 01:43:56.677908 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.677877 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca" pod "a158d236-9b43-4c8a-9203-74f1a3781b76" (UID: "a158d236-9b43-4c8a-9203-74f1a3781b76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:56.680095 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.680063 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca" pod "a158d236-9b43-4c8a-9203-74f1a3781b76" (UID: "a158d236-9b43-4c8a-9203-74f1a3781b76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:43:56.680246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.680220 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz" pod "a158d236-9b43-4c8a-9203-74f1a3781b76" (UID: "a158d236-9b43-4c8a-9203-74f1a3781b76"). InnerVolumeSpecName "kube-api-access-rvmlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:56.778844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.778804 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:56.778844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.778839 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmlz\" (UniqueName: \"kubernetes.io/projected/a158d236-9b43-4c8a-9203-74f1a3781b76-kube-api-access-rvmlz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:56.778844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:56.778849 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a158d236-9b43-4c8a-9203-74f1a3781b76-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:57.360460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:57.360429 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:43:57.423972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:57.423934 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" Nov 04 01:43:57.424176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:57.423974 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb" event={"ID":"a158d236-9b43-4c8a-9203-74f1a3781b76","Type":"ContainerDied","Data":"059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949"} Nov 04 01:43:57.424176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:43:57.424045 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059eee255586fc6ffdf2588b812c9c991c071e5138c7499d94806376627ef949" Nov 04 01:44:10.064892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:10.064827 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb"] Nov 04 01:44:10.112099 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:10.112051 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f9348a-tfvpb"] Nov 04 01:44:11.348510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:11.348477 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a158d236-9b43-4c8a-9203-74f1a3781b76" path="/var/lib/kubelet/pods/a158d236-9b43-4c8a-9203-74f1a3781b76/volumes" Nov 04 01:44:12.381820 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.381749 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" podUID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" containerName="main" containerID="cri-o://be9bde80772cab2f0d37a598618646625df201ee313699899535e8a45e3f5185" gracePeriod=45 Nov 04 01:44:12.522920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.522882 4486 generic.go:356] "Generic (PLEG): container finished" podID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" containerID="be9bde80772cab2f0d37a598618646625df201ee313699899535e8a45e3f5185" exitCode=0 Nov 04 01:44:12.523112 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.522950 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" event={"ID":"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe","Type":"ContainerDied","Data":"be9bde80772cab2f0d37a598618646625df201ee313699899535e8a45e3f5185"} Nov 04 01:44:12.654570 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.654526 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747719 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747785 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747830 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747857 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747922 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\" (UniqueName: \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.747963 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\") pod \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\" (UID: \"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe\") " Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.748769 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.749143 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.749709 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.751099 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.755953 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "kube-api-access-vkx7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.756166 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:12.758025 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.757371 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm" pod "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" (UID: "79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:12.850535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.850504 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:12.850535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.850533 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:12.850535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.850543 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:12.850773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.850554 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkx7f\" (UniqueName: \"kubernetes.io/projected/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-kube-api-access-vkx7f\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:12.850773 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:12.850565 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:13.531448 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:13.531412 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" Nov 04 01:44:13.531966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:13.531451 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t" event={"ID":"79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe","Type":"ContainerDied","Data":"91a42b7d0fb95c7e588232e6da17ec2029ba96810859a0a81341c53efff60ba7"} Nov 04 01:44:13.531966 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:13.531512 4486 scope.go:122] "RemoveContainer" containerID="be9bde80772cab2f0d37a598618646625df201ee313699899535e8a45e3f5185" Nov 04 01:44:13.689497 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:13.689398 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:44:13.736414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:13.736370 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f9348a/simulator-single-main-f9348a-kserve-7c66794755-l8m2t"] Nov 04 01:44:13.831926 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:44:13.831880 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:44:15.347471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:15.347430 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" path="/var/lib/kubelet/pods/79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe/volumes" Nov 04 01:44:16.441585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:16.441545 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:44:16.442374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:16.442337 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" containerID="cri-o://5213494f88323ab5a7804f40b029f4138aed745b99760f7b466895734b253592" gracePeriod=30 Nov 04 01:44:17.983055 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:17.983018 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2"] Nov 04 01:44:17.986082 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:17.986051 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fn9j2"] Nov 04 01:44:18.101959 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.101921 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh"] Nov 04 01:44:18.102434 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102420 4486 cpu_manager.go:414] "containerMap: removing container" podUID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" containerName="main" Nov 04 01:44:18.102498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102435 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" containerName="main" Nov 04 01:44:18.102498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102457 4486 cpu_manager.go:414] "containerMap: removing container" podUID="a158d236-9b43-4c8a-9203-74f1a3781b76" containerName="revoke-keys" Nov 04 01:44:18.102498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102464 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a158d236-9b43-4c8a-9203-74f1a3781b76" containerName="revoke-keys" Nov 04 01:44:18.102599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102528 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f403d7-0ed7-4ce9-aca6-8f3ab0da8dfe" containerName="main" Nov 04 01:44:18.102599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.102540 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="a158d236-9b43-4c8a-9203-74f1a3781b76" containerName="revoke-keys" Nov 04 01:44:18.106867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.106849 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.156898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.156863 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:44:18.157100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.157017 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:44:18.159774 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.159745 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh"] Nov 04 01:44:18.208784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.208747 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.208992 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.208875 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.208992 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.208924 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.309924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.309825 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.309924 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.309914 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.310209 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.309950 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.310726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.310705 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.312435 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.312410 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.366059 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.366019 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\") pod \"maas-api-revoke-sub-3d4156f1-pl4zh\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.419921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.419880 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:18.791864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:18.791823 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh"] Nov 04 01:44:18.793579 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:44:18.793547 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65963ab9_52fd_4ad8_820f_bb1672596211.slice/crio-b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e WatchSource:0}: Error finding container b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e: Status 404 returned error can't find the container with id b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e Nov 04 01:44:19.348457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:19.348426 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa995a60-1848-4b21-9709-bfa6ebae2dd0" path="/var/lib/kubelet/pods/fa995a60-1848-4b21-9709-bfa6ebae2dd0/volumes" Nov 04 01:44:19.574762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:19.574725 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" event={"ID":"65963ab9-52fd-4ad8-820f-bb1672596211","Type":"ContainerStarted","Data":"52de49655507e8ee6e25a9cc538b25ddc743a6048d843e8f03728ddef79332ae"} Nov 04 01:44:19.574762 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:19.574784 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" event={"ID":"65963ab9-52fd-4ad8-820f-bb1672596211","Type":"ContainerStarted","Data":"b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e"} Nov 04 01:44:19.683974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:19.683868 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" podStartSLOduration=1.683853539 podStartE2EDuration="1.683853539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:44:19.682484139 +0000 UTC m=+2828.441146304" watchObservedRunningTime="2026-11-04 01:44:19.683853539 +0000 UTC m=+2828.442515700" Nov 04 01:44:20.582919 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:20.582878 4486 generic.go:356] "Generic (PLEG): container finished" podID="65963ab9-52fd-4ad8-820f-bb1672596211" containerID="52de49655507e8ee6e25a9cc538b25ddc743a6048d843e8f03728ddef79332ae" exitCode=0 Nov 04 01:44:20.583336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:20.582953 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" event={"ID":"65963ab9-52fd-4ad8-820f-bb1672596211","Type":"ContainerDied","Data":"52de49655507e8ee6e25a9cc538b25ddc743a6048d843e8f03728ddef79332ae"} Nov 04 01:44:21.331721 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.331686 4486 scope.go:122] "RemoveContainer" containerID="021bba3bd887e42a1bd282dca824e10f13caa0bac0963c036fe7ae12370a60d9" Nov 04 01:44:21.341265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.341236 4486 scope.go:122] "RemoveContainer" containerID="d124df22263459c7b12adddce1b92ef077ebca0263611b21514fa8246bae1db1" Nov 04 01:44:21.354136 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.354098 4486 scope.go:122] "RemoveContainer" containerID="2caa93454aef36f28c09490050b60342e0284eed043e6f2b02f38ab48ed273bc" Nov 04 01:44:21.594052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.593525 4486 generic.go:356] "Generic (PLEG): container finished" podID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerID="5213494f88323ab5a7804f40b029f4138aed745b99760f7b466895734b253592" exitCode=0 Nov 04 01:44:21.594052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.593807 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerDied","Data":"5213494f88323ab5a7804f40b029f4138aed745b99760f7b466895734b253592"} Nov 04 01:44:21.594052 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.593866 4486 scope.go:122] "RemoveContainer" containerID="303bc63cd5b41fa9ff181f0de86ddee79a8d3c6b7f6f8010f3e28fa079596080" Nov 04 01:44:21.721488 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.721456 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:44:21.769800 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.769769 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:21.846935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.846852 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.846935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.846901 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\") pod \"65963ab9-52fd-4ad8-820f-bb1672596211\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.846978 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847022 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847082 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\") pod \"65963ab9-52fd-4ad8-820f-bb1672596211\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847113 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847173 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\") pod \"65963ab9-52fd-4ad8-820f-bb1672596211\" (UID: \"65963ab9-52fd-4ad8-820f-bb1672596211\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847184 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847224 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847303 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847291 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847332 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847371 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847402 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\") pod \"90dd3754-5fae-4d7c-ab93-349f83bf4017\" (UID: \"90dd3754-5fae-4d7c-ab93-349f83bf4017\") " Nov 04 01:44:21.847680 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847464 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:21.847905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847726 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:21.847905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847829 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.847905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847852 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.847905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.847868 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.848158 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.848097 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:21.848905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.848629 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:21.848905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.848683 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca" pod "65963ab9-52fd-4ad8-820f-bb1672596211" (UID: "65963ab9-52fd-4ad8-820f-bb1672596211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:44:21.850142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.850108 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca" pod "65963ab9-52fd-4ad8-820f-bb1672596211" (UID: "65963ab9-52fd-4ad8-820f-bb1672596211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:44:21.850232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.850194 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "kube-api-access-vkmpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:21.850359 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.850330 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:21.851302 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.851265 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q" pod "65963ab9-52fd-4ad8-820f-bb1672596211" (UID: "65963ab9-52fd-4ad8-820f-bb1672596211"). InnerVolumeSpecName "kube-api-access-c4t9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:21.851957 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.851934 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:44:21.852420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.852398 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy" pod "90dd3754-5fae-4d7c-ab93-349f83bf4017" (UID: "90dd3754-5fae-4d7c-ab93-349f83bf4017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:21.949120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949081 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949121 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65963ab9-52fd-4ad8-820f-bb1672596211-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949136 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949152 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4t9q\" (UniqueName: \"kubernetes.io/projected/65963ab9-52fd-4ad8-820f-bb1672596211-kube-api-access-c4t9q\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949166 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949183 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90dd3754-5fae-4d7c-ab93-349f83bf4017-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949197 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949211 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkmpx\" (UniqueName: \"kubernetes.io/projected/90dd3754-5fae-4d7c-ab93-349f83bf4017-kube-api-access-vkmpx\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:21.949348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:21.949225 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90dd3754-5fae-4d7c-ab93-349f83bf4017-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:22.603120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.603086 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" Nov 04 01:44:22.603120 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.603111 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq" event={"ID":"90dd3754-5fae-4d7c-ab93-349f83bf4017","Type":"ContainerDied","Data":"f4bd82303030661a3c11b8947aef89906c74c764c61c151f1aea7798f3f66cc0"} Nov 04 01:44:22.603682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.603207 4486 scope.go:122] "RemoveContainer" containerID="5213494f88323ab5a7804f40b029f4138aed745b99760f7b466895734b253592" Nov 04 01:44:22.605161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.604685 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" Nov 04 01:44:22.605161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.604720 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh" event={"ID":"65963ab9-52fd-4ad8-820f-bb1672596211","Type":"ContainerDied","Data":"b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e"} Nov 04 01:44:22.605161 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.604762 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b471f661d65e741cf9ed1c656db531838545f1a0b894c56432a04f9bf4d58e" Nov 04 01:44:22.821631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.821595 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:44:22.833753 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:22.833720 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f9348a-openshift-default-74c68f958f6vtnq"] Nov 04 01:44:23.348498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:23.348453 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" path="/var/lib/kubelet/pods/90dd3754-5fae-4d7c-ab93-349f83bf4017/volumes" Nov 04 01:44:26.053309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:26.053271 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb"] Nov 04 01:44:26.100560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:26.100521 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wsjpb"] Nov 04 01:44:27.348065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:27.348025 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27df86f7-8441-4e0d-9f73-7fbbc3a28aaa" path="/var/lib/kubelet/pods/27df86f7-8441-4e0d-9f73-7fbbc3a28aaa/volumes" Nov 04 01:44:36.815903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:36.815868 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:44:36.816399 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:36.816106 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" podUID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" containerName="manager" containerID="cri-o://3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c" gracePeriod=10 Nov 04 01:44:37.102124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.102095 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:44:37.202198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.202162 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\" (UniqueName: \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\") pod \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " Nov 04 01:44:37.202198 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.202203 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\") pod \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\" (UID: \"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e\") " Nov 04 01:44:37.202658 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.202636 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume" pod "c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" (UID: "c808b6f0-06f4-4ba4-8c82-0b650bb4c99e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:44:37.204499 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.204470 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877" pod "c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" (UID: "c808b6f0-06f4-4ba4-8c82-0b650bb4c99e"). InnerVolumeSpecName "kube-api-access-kv877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:44:37.303519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.303484 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv877\" (UniqueName: \"kubernetes.io/projected/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-kube-api-access-kv877\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:37.303519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.303515 4486 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e-extensions-socket-volume\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:44:37.696251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.696155 4486 generic.go:356] "Generic (PLEG): container finished" podID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" containerID="3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c" exitCode=0 Nov 04 01:44:37.696251 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.696207 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" Nov 04 01:44:37.696472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.696238 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" event={"ID":"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e","Type":"ContainerDied","Data":"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c"} Nov 04 01:44:37.696472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.696314 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx" event={"ID":"c808b6f0-06f4-4ba4-8c82-0b650bb4c99e","Type":"ContainerDied","Data":"b02f91774a8efe33b6507d72da0d661b4c618acd0486445f45df78255a2bbed8"} Nov 04 01:44:37.696472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.696339 4486 scope.go:122] "RemoveContainer" containerID="3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c" Nov 04 01:44:37.707482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.707458 4486 scope.go:122] "RemoveContainer" containerID="3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c" Nov 04 01:44:37.707749 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:44:37.707730 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c\": container with ID starting with 3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c not found: ID does not exist" containerID="3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c" Nov 04 01:44:37.707802 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.707759 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c"} err="failed to get container status \"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c\": rpc error: code = NotFound desc = could not find container \"3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c\": container with ID starting with 3049eda918b7acc58b08c8856b39a2f5bc9af09c6a8a7807783886abdf9fa80c not found: ID does not exist" Nov 04 01:44:37.901290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.901246 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:44:37.906272 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:37.906243 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7zghx"] Nov 04 01:44:39.347606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:44:39.347570 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" path="/var/lib/kubelet/pods/c808b6f0-06f4-4ba4-8c82-0b650bb4c99e/volumes" Nov 04 01:44:43.831556 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:44:43.831514 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:45:00.053501 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.053460 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp"] Nov 04 01:45:00.100673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.100636 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fsmp"] Nov 04 01:45:00.162621 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.162580 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:00.163053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163031 4486 cpu_manager.go:414] "containerMap: removing container" podUID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" containerName="manager" Nov 04 01:45:00.163053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163053 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" containerName="manager" Nov 04 01:45:00.163159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163074 4486 cpu_manager.go:414] "containerMap: removing container" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:00.163159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163079 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:00.163159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163107 4486 cpu_manager.go:414] "containerMap: removing container" podUID="65963ab9-52fd-4ad8-820f-bb1672596211" containerName="revoke-keys" Nov 04 01:45:00.163159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163112 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65963ab9-52fd-4ad8-820f-bb1672596211" containerName="revoke-keys" Nov 04 01:45:00.163317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163172 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:00.163317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163180 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="65963ab9-52fd-4ad8-820f-bb1672596211" containerName="revoke-keys" Nov 04 01:45:00.163317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163188 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="c808b6f0-06f4-4ba4-8c82-0b650bb4c99e" containerName="manager" Nov 04 01:45:00.163317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.163196 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:00.166617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.166598 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.217340 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.217289 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:45:00.220197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.220163 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:00.228888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.228850 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqz2\" (UniqueName: \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.229084 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.229042 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.329793 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.329756 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.329969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.329855 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqz2\" (UniqueName: \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.330480 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.330459 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.384720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.384684 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqz2\" (UniqueName: \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\") pod \"maas-api-key-cleanup-29895945-5f6zp\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.479548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.479513 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:00.849018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:00.848978 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:00.850609 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:45:00.850579 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27b5cca_eab4_413e_876d_93e5e6c746f2.slice/crio-576cad5624072482ece5728788923da8d5499097f4bfa4f570e9751f1fecdefc WatchSource:0}: Error finding container 576cad5624072482ece5728788923da8d5499097f4bfa4f570e9751f1fecdefc: Status 404 returned error can't find the container with id 576cad5624072482ece5728788923da8d5499097f4bfa4f570e9751f1fecdefc Nov 04 01:45:01.348114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:01.348077 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666d71b5-d4ab-4c77-9360-ccba4b86981d" path="/var/lib/kubelet/pods/666d71b5-d4ab-4c77-9360-ccba4b86981d/volumes" Nov 04 01:45:01.852665 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:01.852620 4486 generic.go:356] "Generic (PLEG): container finished" podID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerID="7bf1879664923a59fa20dc2d551d2fc6fc8c408f6b9af896995e5c7e007ba488" exitCode=60 Nov 04 01:45:01.852892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:01.852702 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerDied","Data":"7bf1879664923a59fa20dc2d551d2fc6fc8c408f6b9af896995e5c7e007ba488"} Nov 04 01:45:01.852892 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:01.852763 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerStarted","Data":"576cad5624072482ece5728788923da8d5499097f4bfa4f570e9751f1fecdefc"} Nov 04 01:45:01.853042 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:01.853004 4486 scope.go:122] "RemoveContainer" containerID="7bf1879664923a59fa20dc2d551d2fc6fc8c408f6b9af896995e5c7e007ba488" Nov 04 01:45:02.861298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:02.861262 4486 generic.go:356] "Generic (PLEG): container finished" podID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerID="a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704" exitCode=60 Nov 04 01:45:02.861750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:02.861358 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerDied","Data":"a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704"} Nov 04 01:45:02.861750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:02.861417 4486 scope.go:122] "RemoveContainer" containerID="7bf1879664923a59fa20dc2d551d2fc6fc8c408f6b9af896995e5c7e007ba488" Nov 04 01:45:02.861750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:02.861614 4486 scope.go:122] "RemoveContainer" containerID="a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704" Nov 04 01:45:02.861926 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:45:02.861901 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-5f6zp_odh-ai-gateway-infra(b27b5cca-eab4-413e-876d-93e5e6c746f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" Nov 04 01:45:03.869363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:03.869333 4486 scope.go:122] "RemoveContainer" containerID="a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704" Nov 04 01:45:03.869787 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:45:03.869557 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-5f6zp_odh-ai-gateway-infra(b27b5cca-eab4-413e-876d-93e5e6c746f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" Nov 04 01:45:13.342559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:13.342526 4486 scope.go:122] "RemoveContainer" containerID="a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704" Nov 04 01:45:13.831236 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:45:13.831197 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:45:13.931645 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:13.931555 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerStarted","Data":"fafaaa810551e26ec4b4ec3fcf5ee6ab94a05c3985271b0ff4f7270f36b59481"} Nov 04 01:45:14.064828 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:14.064774 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" podStartSLOduration=14.064759919 podStartE2EDuration="14.064759919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:14.063054713 +0000 UTC m=+2882.821716873" watchObservedRunningTime="2026-11-04 01:45:14.064759919 +0000 UTC m=+2882.823422079" Nov 04 01:45:14.938887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:14.938854 4486 generic.go:356] "Generic (PLEG): container finished" podID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerID="fafaaa810551e26ec4b4ec3fcf5ee6ab94a05c3985271b0ff4f7270f36b59481" exitCode=60 Nov 04 01:45:14.939346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:14.938929 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerDied","Data":"fafaaa810551e26ec4b4ec3fcf5ee6ab94a05c3985271b0ff4f7270f36b59481"} Nov 04 01:45:14.939346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:14.939016 4486 scope.go:122] "RemoveContainer" containerID="a173f75e27cbfd11d99e3eb0baa37adbf334b6e23856273781219f9073857704" Nov 04 01:45:14.939460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:14.939387 4486 scope.go:122] "RemoveContainer" containerID="fafaaa810551e26ec4b4ec3fcf5ee6ab94a05c3985271b0ff4f7270f36b59481" Nov 04 01:45:14.939699 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:45:14.939676 4486 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895945-5f6zp_odh-ai-gateway-infra(b27b5cca-eab4-413e-876d-93e5e6c746f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" Nov 04 01:45:15.073864 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:15.073825 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:16.091363 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.091326 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:16.191487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.191446 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\" (UniqueName: \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\") pod \"b27b5cca-eab4-413e-876d-93e5e6c746f2\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " Nov 04 01:45:16.191672 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.191539 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\") pod \"b27b5cca-eab4-413e-876d-93e5e6c746f2\" (UID: \"b27b5cca-eab4-413e-876d-93e5e6c746f2\") " Nov 04 01:45:16.192327 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.192270 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca" pod "b27b5cca-eab4-413e-876d-93e5e6c746f2" (UID: "b27b5cca-eab4-413e-876d-93e5e6c746f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:16.194037 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.192785 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27b5cca-eab4-413e-876d-93e5e6c746f2-maas-api-service-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:16.194602 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.194571 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2" pod "b27b5cca-eab4-413e-876d-93e5e6c746f2" (UID: "b27b5cca-eab4-413e-876d-93e5e6c746f2"). InnerVolumeSpecName "kube-api-access-6tqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:16.293949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.293912 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tqz2\" (UniqueName: \"kubernetes.io/projected/b27b5cca-eab4-413e-876d-93e5e6c746f2-kube-api-access-6tqz2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:16.954545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.954509 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" event={"ID":"b27b5cca-eab4-413e-876d-93e5e6c746f2","Type":"ContainerDied","Data":"576cad5624072482ece5728788923da8d5499097f4bfa4f570e9751f1fecdefc"} Nov 04 01:45:16.954545 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.954541 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp" Nov 04 01:45:16.954805 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:16.954578 4486 scope.go:122] "RemoveContainer" containerID="fafaaa810551e26ec4b4ec3fcf5ee6ab94a05c3985271b0ff4f7270f36b59481" Nov 04 01:45:17.112211 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:17.112173 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:17.159292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:17.159255 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895945-5f6zp"] Nov 04 01:45:17.348121 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:17.348080 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" path="/var/lib/kubelet/pods/b27b5cca-eab4-413e-876d-93e5e6c746f2/volumes" Nov 04 01:45:21.053791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:21.053745 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97"] Nov 04 01:45:21.057720 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:21.057680 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fsm97"] Nov 04 01:45:21.355469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:21.355430 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48d68c8-9f6f-4e5f-84e1-05a72d2eac44" path="/var/lib/kubelet/pods/c48d68c8-9f6f-4e5f-84e1-05a72d2eac44/volumes" Nov 04 01:45:21.395601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:21.395567 4486 scope.go:122] "RemoveContainer" containerID="70d1ff99d71700ce3d66a5e5991d1a9cdcd39c0c7316c7e9ae3768678b160f46" Nov 04 01:45:40.451791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.451755 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b"] Nov 04 01:45:40.452423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452402 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452428 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452455 4486 cpu_manager.go:414] "containerMap: removing container" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:40.452472 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452460 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90dd3754-5fae-4d7c-ab93-349f83bf4017" containerName="istio-proxy" Nov 04 01:45:40.452576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452479 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452485 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452570 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452581 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.452673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.452588 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:45:40.456105 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.456080 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.490047 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.490014 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b"] Nov 04 01:45:40.534576 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.534541 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jrbsm\"" Nov 04 01:45:40.543553 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.543518 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.543744 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.543688 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfs7s\" (UniqueName: \"kubernetes.io/projected/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-kube-api-access-qfs7s\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.644896 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.644845 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfs7s\" (UniqueName: \"kubernetes.io/projected/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-kube-api-access-qfs7s\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.645134 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.644947 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.645348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.645329 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.700729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.700690 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfs7s\" (UniqueName: \"kubernetes.io/projected/fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3-kube-api-access-qfs7s\") pod \"kuadrant-operator-controller-manager-678d87766f-6pt7b\" (UID: \"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:40.769333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:40.769239 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:41.210232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:41.210201 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b"] Nov 04 01:45:41.212426 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:45:41.212394 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3d7323_ac5b_4ade_bc40_09b0ba4acdf3.slice/crio-0b9954eed63331ed05b422246c5fe856161cf4988dcbead20929fe66486efd41 WatchSource:0}: Error finding container 0b9954eed63331ed05b422246c5fe856161cf4988dcbead20929fe66486efd41: Status 404 returned error can't find the container with id 0b9954eed63331ed05b422246c5fe856161cf4988dcbead20929fe66486efd41 Nov 04 01:45:42.109674 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:42.109628 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" event={"ID":"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3","Type":"ContainerStarted","Data":"7cca1c25339c95f3058a1a80f80ed533ffa86298ede3b1ca6e6b7a6e5ce51ab7"} Nov 04 01:45:42.110220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:42.109696 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" event={"ID":"fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3","Type":"ContainerStarted","Data":"0b9954eed63331ed05b422246c5fe856161cf4988dcbead20929fe66486efd41"} Nov 04 01:45:42.110220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:42.109739 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:45:42.333010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:42.332924 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" podStartSLOduration=2.332908054 podStartE2EDuration="2.332908054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:42.283315074 +0000 UTC m=+2911.041977234" watchObservedRunningTime="2026-11-04 01:45:42.332908054 +0000 UTC m=+2911.091570215" Nov 04 01:45:43.831496 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:45:43.831458 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:45:53.118960 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:45:53.118883 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6pt7b" Nov 04 01:46:06.760689 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.760644 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d"] Nov 04 01:46:06.761164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.761147 4486 cpu_manager.go:414] "containerMap: removing container" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:46:06.761208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.761165 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27b5cca-eab4-413e-876d-93e5e6c746f2" containerName="cleanup" Nov 04 01:46:06.764606 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.764584 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.826220 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.826179 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2z5j\" (UniqueName: \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.826439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.826320 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.826439 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.826351 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.861610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.861569 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:06.861951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.861926 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:46:06.872324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.866064 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d"] Nov 04 01:46:06.927228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.927192 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.927228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.927233 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.927452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.927287 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2z5j\" (UniqueName: \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.927935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.927913 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.929724 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.929690 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:06.983750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:06.983719 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2z5j\" (UniqueName: \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\") pod \"maas-api-revoke-sub-d576e68b-7rv4d\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:07.076940 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:07.076906 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:07.448043 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:07.448014 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d"] Nov 04 01:46:07.449397 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:46:07.449367 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df29f09_f5c9_419c_98cc_4c73f7df2c95.slice/crio-ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb WatchSource:0}: Error finding container ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb: Status 404 returned error can't find the container with id ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb Nov 04 01:46:08.315542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:08.315501 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" event={"ID":"1df29f09-f5c9-419c-98cc-4c73f7df2c95","Type":"ContainerStarted","Data":"d9b15e32068ca72b32d4ef4e8ebca07c5513bc33acaab372d35ad19a22ca71c0"} Nov 04 01:46:08.316006 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:08.315564 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" event={"ID":"1df29f09-f5c9-419c-98cc-4c73f7df2c95","Type":"ContainerStarted","Data":"ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb"} Nov 04 01:46:08.427761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:08.427703 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" podStartSLOduration=2.427684637 podStartE2EDuration="2.427684637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:08.424806693 +0000 UTC m=+2937.183468836" watchObservedRunningTime="2026-11-04 01:46:08.427684637 +0000 UTC m=+2937.186346836" Nov 04 01:46:09.323887 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:09.323850 4486 generic.go:356] "Generic (PLEG): container finished" podID="1df29f09-f5c9-419c-98cc-4c73f7df2c95" containerID="d9b15e32068ca72b32d4ef4e8ebca07c5513bc33acaab372d35ad19a22ca71c0" exitCode=0 Nov 04 01:46:09.324323 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:09.323935 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" event={"ID":"1df29f09-f5c9-419c-98cc-4c73f7df2c95","Type":"ContainerDied","Data":"d9b15e32068ca72b32d4ef4e8ebca07c5513bc33acaab372d35ad19a22ca71c0"} Nov 04 01:46:10.486033 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.486007 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:10.563827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.563792 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\") pod \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " Nov 04 01:46:10.564035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.563862 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\" (UniqueName: \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\") pod \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " Nov 04 01:46:10.564035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.563900 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\") pod \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\" (UID: \"1df29f09-f5c9-419c-98cc-4c73f7df2c95\") " Nov 04 01:46:10.564268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.564236 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca" pod "1df29f09-f5c9-419c-98cc-4c73f7df2c95" (UID: "1df29f09-f5c9-419c-98cc-4c73f7df2c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:10.564349 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.564327 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:10.566124 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.566093 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca" pod "1df29f09-f5c9-419c-98cc-4c73f7df2c95" (UID: "1df29f09-f5c9-419c-98cc-4c73f7df2c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:10.566249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.566155 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j" pod "1df29f09-f5c9-419c-98cc-4c73f7df2c95" (UID: "1df29f09-f5c9-419c-98cc-4c73f7df2c95"). InnerVolumeSpecName "kube-api-access-p2z5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:10.665295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.665204 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2z5j\" (UniqueName: \"kubernetes.io/projected/1df29f09-f5c9-419c-98cc-4c73f7df2c95-kube-api-access-p2z5j\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:10.665295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:10.665239 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df29f09-f5c9-419c-98cc-4c73f7df2c95-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:11.336895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:11.336857 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" event={"ID":"1df29f09-f5c9-419c-98cc-4c73f7df2c95","Type":"ContainerDied","Data":"ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb"} Nov 04 01:46:11.337128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:11.336907 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace0f6f29599a1ce877edc3bd4ebaf62197d989f7ca4fcfde8dec3eda0990efb" Nov 04 01:46:11.337128 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:11.336869 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d" Nov 04 01:46:13.831682 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:46:13.831644 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:46:21.433385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:21.433345 4486 scope.go:122] "RemoveContainer" containerID="4b7a98f3b256f5d68606ca37a905716bbd45f8bb70060e7809e745dc45df6b8c" Nov 04 01:46:21.443159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:21.443131 4486 scope.go:122] "RemoveContainer" containerID="41c0eae3f8d368188cdef317bb802d7cbeaa51c2d2e1349616c2e18831c5b9a1" Nov 04 01:46:21.452166 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:21.452140 4486 scope.go:122] "RemoveContainer" containerID="11a114203287fd4c4f2eb6927f9ad89da98e2a65d42cb0ddcdf89802a296095e" Nov 04 01:46:31.054620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:31.054580 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln"] Nov 04 01:46:31.062249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:31.062219 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9sjln"] Nov 04 01:46:31.348018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:31.347969 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3861eebd-ba5d-4626-be76-2c09b4ddd78f" path="/var/lib/kubelet/pods/3861eebd-ba5d-4626-be76-2c09b4ddd78f/volumes" Nov 04 01:46:35.806997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.806948 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5"] Nov 04 01:46:35.807642 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.807625 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1df29f09-f5c9-419c-98cc-4c73f7df2c95" containerName="revoke-keys" Nov 04 01:46:35.807690 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.807644 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df29f09-f5c9-419c-98cc-4c73f7df2c95" containerName="revoke-keys" Nov 04 01:46:35.807761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.807751 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df29f09-f5c9-419c-98cc-4c73f7df2c95" containerName="revoke-keys" Nov 04 01:46:35.812972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.812950 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:35.865347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.865307 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5"] Nov 04 01:46:35.925032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.924970 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:35.925237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.925168 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:46:35.956130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.956093 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:35.956319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.956156 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7bz\" (UniqueName: \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:35.956319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:35.956223 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.057438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.057348 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7bz\" (UniqueName: \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.057438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.057400 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.057644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.057482 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.058125 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.058098 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.059913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.059887 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.111577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.111544 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7bz\" (UniqueName: \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\") pod \"maas-api-revoke-sub-d2bc93bc-ld8f5\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.130421 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.130385 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:36.501918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.501884 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5"] Nov 04 01:46:36.503960 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:46:36.503929 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350fb998_9caf_4c45_9d9d_aac988c53bb1.slice/crio-b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99 WatchSource:0}: Error finding container b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99: Status 404 returned error can't find the container with id b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99 Nov 04 01:46:36.512620 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:36.512589 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" event={"ID":"350fb998-9caf-4c45-9d9d-aac988c53bb1","Type":"ContainerStarted","Data":"b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99"} Nov 04 01:46:37.520345 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:37.520309 4486 generic.go:356] "Generic (PLEG): container finished" podID="350fb998-9caf-4c45-9d9d-aac988c53bb1" containerID="07b462ec8e72e0ee501ef1c693c0d94642386da3f91b1ebc27f1986a63cfc257" exitCode=0 Nov 04 01:46:37.520767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:37.520397 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" event={"ID":"350fb998-9caf-4c45-9d9d-aac988c53bb1","Type":"ContainerDied","Data":"07b462ec8e72e0ee501ef1c693c0d94642386da3f91b1ebc27f1986a63cfc257"} Nov 04 01:46:38.671622 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.671595 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:38.785329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.785236 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\") pod \"350fb998-9caf-4c45-9d9d-aac988c53bb1\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " Nov 04 01:46:38.785329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.785279 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\") pod \"350fb998-9caf-4c45-9d9d-aac988c53bb1\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " Nov 04 01:46:38.785329 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.785328 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\" (UniqueName: \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\") pod \"350fb998-9caf-4c45-9d9d-aac988c53bb1\" (UID: \"350fb998-9caf-4c45-9d9d-aac988c53bb1\") " Nov 04 01:46:38.785709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.785685 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca" pod "350fb998-9caf-4c45-9d9d-aac988c53bb1" (UID: "350fb998-9caf-4c45-9d9d-aac988c53bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:38.787508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.787478 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca" pod "350fb998-9caf-4c45-9d9d-aac988c53bb1" (UID: "350fb998-9caf-4c45-9d9d-aac988c53bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:38.787655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.787609 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz" pod "350fb998-9caf-4c45-9d9d-aac988c53bb1" (UID: "350fb998-9caf-4c45-9d9d-aac988c53bb1"). InnerVolumeSpecName "kube-api-access-xq7bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:38.886767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.886736 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.886767 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.886763 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/350fb998-9caf-4c45-9d9d-aac988c53bb1-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:38.886978 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:38.886775 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7bz\" (UniqueName: \"kubernetes.io/projected/350fb998-9caf-4c45-9d9d-aac988c53bb1-kube-api-access-xq7bz\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:39.533749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:39.533720 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" Nov 04 01:46:39.533961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:39.533718 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5" event={"ID":"350fb998-9caf-4c45-9d9d-aac988c53bb1","Type":"ContainerDied","Data":"b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99"} Nov 04 01:46:39.533961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:39.533845 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b45193d106a6aca76a423673dae847f08d823f9f3265bac6459ef73daab60a99" Nov 04 01:46:42.048332 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:42.048292 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f"] Nov 04 01:46:42.095394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:42.095346 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-tjw5f"] Nov 04 01:46:43.358414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:43.358378 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5" path="/var/lib/kubelet/pods/36b55bc3-7d0b-467b-9b50-c6b5bb8c50f5/volumes" Nov 04 01:46:43.832169 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:46:43.832121 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:46:48.718085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.718040 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:46:48.718750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.718728 4486 cpu_manager.go:414] "containerMap: removing container" podUID="350fb998-9caf-4c45-9d9d-aac988c53bb1" containerName="revoke-keys" Nov 04 01:46:48.718750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.718750 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="350fb998-9caf-4c45-9d9d-aac988c53bb1" containerName="revoke-keys" Nov 04 01:46:48.718900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.718843 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="350fb998-9caf-4c45-9d9d-aac988c53bb1" containerName="revoke-keys" Nov 04 01:46:48.722423 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.722402 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.775965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.775934 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f4981c-openshift-default-dockercfg-2tgzl\"" Nov 04 01:46:48.779290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.779253 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.779640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.779611 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.779849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.779833 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.780019 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.780001 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.780347 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.780273 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55qs\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.780591 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.780572 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.780814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.780796 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.781032 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.781014 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.781244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.781215 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.823317 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.823278 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:46:48.882268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882217 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882282 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882325 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882368 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882414 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882447 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882483 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882471 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882513 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55qs\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882600 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.882814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882666 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.883024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.882895 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.883141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.883116 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.883388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.883363 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.883506 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.883481 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.885751 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.885719 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.885975 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.885953 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.937898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.937856 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:48.938110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:48.938044 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55qs\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\") pod \"e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:49.039201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.039103 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:49.486133 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.486095 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:46:49.523566 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:46:49.523531 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd138dd76_3ddd_4ed3_ada5_2d1f164321a1.slice/crio-97747e552da196dbf349d03d8a05182b0ca2f2498272f9a8243a738493e3b8e1 WatchSource:0}: Error finding container 97747e552da196dbf349d03d8a05182b0ca2f2498272f9a8243a738493e3b8e1: Status 404 returned error can't find the container with id 97747e552da196dbf349d03d8a05182b0ca2f2498272f9a8243a738493e3b8e1 Nov 04 01:46:49.525697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.525651 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:46:49.525824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.525748 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:46:49.525824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.525794 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:46:49.596609 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.596569 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerStarted","Data":"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607"} Nov 04 01:46:49.596815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.596643 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerStarted","Data":"97747e552da196dbf349d03d8a05182b0ca2f2498272f9a8243a738493e3b8e1"} Nov 04 01:46:49.711784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:49.711725 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podStartSLOduration=1.711708754 podStartE2EDuration="1.711708754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:49.708560354 +0000 UTC m=+2978.467222515" watchObservedRunningTime="2026-11-04 01:46:49.711708754 +0000 UTC m=+2978.470370914" Nov 04 01:46:50.039854 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:50.039815 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:50.045540 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:50.045510 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:50.602630 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:50.602599 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:46:50.603512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:46:50.603490 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:47:13.835132 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:47:13.835062 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:47:16.810153 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.810120 4486 generic.go:356] "Generic (PLEG): container finished" podID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerID="e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607" exitCode=0 Nov 04 01:47:16.810667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.810197 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerDied","Data":"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607"} Nov 04 01:47:16.810667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.810564 4486 scope.go:122] "RemoveContainer" containerID="e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607" Nov 04 01:47:16.811801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.811774 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:47:16.811903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.811836 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:47:16.811903 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:16.811868 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:47:17.819056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:17.819018 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerStarted","Data":"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed"} Nov 04 01:47:18.042109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.042055 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:47:18.042404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.042371 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 04 01:47:18.042503 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.042436 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 04 01:47:18.100750 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.100661 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:47:18.116749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.116714 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:47:18.140007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.128094 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:47:18.169786 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:18.169758 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:47:19.051015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:19.050560 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 04 01:47:19.051015 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:19.050642 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 04 01:47:20.042624 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.042584 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 04 01:47:20.043100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.043062 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 04 01:47:20.603360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.603315 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:47:20.854605 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.854521 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:47:20.858787 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.858756 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894095 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hj9\" (UniqueName: \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894178 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894241 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894293 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894323 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.894461 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.894376 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:20.913558 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.913520 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:47:20.955709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.955672 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f4981c\"/\"openshift-service-ca.crt\"" Nov 04 01:47:20.956010 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.955878 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f4981c\"/\"default-dockercfg-475mv\"" Nov 04 01:47:20.956115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.956024 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f4981c\"/\"simulator-single-main-f4981c-kserve-self-signed-certs\"" Nov 04 01:47:20.956115 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.956083 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f4981c\"/\"kube-root-ca.crt\"" Nov 04 01:47:20.998398 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998354 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998406 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998475 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998505 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998574 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:20.998726 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hj9\" (UniqueName: \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.000016 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.000188 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.000535 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.000248 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.009962 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.009928 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.010530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.010498 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.050536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.050491 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 04 01:47:21.050832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.050804 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 04 01:47:21.055963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.055933 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hj9\" (UniqueName: \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\") pod \"simulator-single-main-f4981c-kserve-775595c45-kk54k\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.173228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.173137 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:21.482554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.482471 4486 scope.go:122] "RemoveContainer" containerID="023c629758bf5ae4078f11fb1397ed3502354a9ed668bce469e9ebdcfc30abda" Nov 04 01:47:21.491905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.491882 4486 scope.go:122] "RemoveContainer" containerID="91ff0dfd9ea0c5f0ad4f6f1d61781cb4e7193d66160a5957f59cf0b251897ad1" Nov 04 01:47:21.575872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.575834 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:47:21.576718 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:47:21.576695 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaf87389_7c4a_4a38_852a_afae769f51fc.slice/crio-8fd88b7c118186dba35bf3ce7bd26f3480ab87e69f2a24979dd66eacdf7624c7 WatchSource:0}: Error finding container 8fd88b7c118186dba35bf3ce7bd26f3480ab87e69f2a24979dd66eacdf7624c7: Status 404 returned error can't find the container with id 8fd88b7c118186dba35bf3ce7bd26f3480ab87e69f2a24979dd66eacdf7624c7 Nov 04 01:47:21.869109 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.869070 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" event={"ID":"caf87389-7c4a-4a38-852a-afae769f51fc","Type":"ContainerStarted","Data":"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99"} Nov 04 01:47:21.869631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.869119 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" event={"ID":"caf87389-7c4a-4a38-852a-afae769f51fc","Type":"ContainerStarted","Data":"8fd88b7c118186dba35bf3ce7bd26f3480ab87e69f2a24979dd66eacdf7624c7"} Nov 04 01:47:21.998322 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:21.998259 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" podStartSLOduration=1.998242393 podStartE2EDuration="1.998242393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:21.995995156 +0000 UTC m=+3010.754657310" watchObservedRunningTime="2026-11-04 01:47:21.998242393 +0000 UTC m=+3010.756904555" Nov 04 01:47:22.045007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:22.044418 4486 patch_prober.go:29] interesting pod/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 04 01:47:22.045007 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:22.044484 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 04 01:47:22.174355 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:22.174257 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:23.044617 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:23.044526 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:47:23.045700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:23.045677 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:47:31.173525 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:31.173483 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:31.186280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:31.186248 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:31.951519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:47:31.951488 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:47:43.832523 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:47:43.832483 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:48:06.330837 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.330795 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl"] Nov 04 01:48:06.335487 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.335464 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.388939 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.388898 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl"] Nov 04 01:48:06.432406 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.432359 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:06.432595 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.432526 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:48:06.501335 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.501289 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.501543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.501456 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.501543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.501516 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c746\" (UniqueName: \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.603254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.603152 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.603254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.603222 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c746\" (UniqueName: \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.603528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.603279 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.603997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.603956 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.605611 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.605592 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.658169 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.658139 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c746\" (UniqueName: \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\") pod \"maas-api-revoke-sub-ab3c5056-8gxgl\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:06.950668 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:06.950571 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:07.333094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:07.333063 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl"] Nov 04 01:48:07.334682 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:48:07.334652 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2bcbe1_d138_40c3_bf36_58c3e04cb026.slice/crio-b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325 WatchSource:0}: Error finding container b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325: Status 404 returned error can't find the container with id b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325 Nov 04 01:48:08.174961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:08.174914 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" event={"ID":"bb2bcbe1-d138-40c3-bf36-58c3e04cb026","Type":"ContainerStarted","Data":"831570df876a1546e905ff505430b5ed465d7baca67474f4515cf3f48baf86fa"} Nov 04 01:48:08.175164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:08.175003 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" event={"ID":"bb2bcbe1-d138-40c3-bf36-58c3e04cb026","Type":"ContainerStarted","Data":"b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325"} Nov 04 01:48:08.287024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:08.286945 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" podStartSLOduration=2.286929883 podStartE2EDuration="2.286929883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:08.284131695 +0000 UTC m=+3057.042793857" watchObservedRunningTime="2026-11-04 01:48:08.286929883 +0000 UTC m=+3057.045592044" Nov 04 01:48:09.214872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:09.214831 4486 generic.go:356] "Generic (PLEG): container finished" podID="bb2bcbe1-d138-40c3-bf36-58c3e04cb026" containerID="831570df876a1546e905ff505430b5ed465d7baca67474f4515cf3f48baf86fa" exitCode=0 Nov 04 01:48:09.215906 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:09.215764 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" event={"ID":"bb2bcbe1-d138-40c3-bf36-58c3e04cb026","Type":"ContainerDied","Data":"831570df876a1546e905ff505430b5ed465d7baca67474f4515cf3f48baf86fa"} Nov 04 01:48:10.362612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.362577 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:10.550601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.550515 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\") pod \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " Nov 04 01:48:10.550758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.550635 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\" (UniqueName: \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\") pod \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " Nov 04 01:48:10.550758 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.550677 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\") pod \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\" (UID: \"bb2bcbe1-d138-40c3-bf36-58c3e04cb026\") " Nov 04 01:48:10.551085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.551060 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca" pod "bb2bcbe1-d138-40c3-bf36-58c3e04cb026" (UID: "bb2bcbe1-d138-40c3-bf36-58c3e04cb026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:10.552781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.552761 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746" pod "bb2bcbe1-d138-40c3-bf36-58c3e04cb026" (UID: "bb2bcbe1-d138-40c3-bf36-58c3e04cb026"). InnerVolumeSpecName "kube-api-access-4c746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:10.552866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.552845 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca" pod "bb2bcbe1-d138-40c3-bf36-58c3e04cb026" (UID: "bb2bcbe1-d138-40c3-bf36-58c3e04cb026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:10.651590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.651546 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:10.651590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.651582 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:10.651590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:10.651594 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c746\" (UniqueName: \"kubernetes.io/projected/bb2bcbe1-d138-40c3-bf36-58c3e04cb026-kube-api-access-4c746\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:11.228342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:11.228313 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" Nov 04 01:48:11.228554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:11.228354 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl" event={"ID":"bb2bcbe1-d138-40c3-bf36-58c3e04cb026","Type":"ContainerDied","Data":"b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325"} Nov 04 01:48:11.228554 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:11.228412 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03c0b1f8343ba585f9c4e6e99883f84b4d3c7622d7e79cd3824e49932bdb325" Nov 04 01:48:13.832192 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:48:13.832139 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:48:17.922848 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.922808 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d"] Nov 04 01:48:17.923529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.923500 4486 cpu_manager.go:414] "containerMap: removing container" podUID="bb2bcbe1-d138-40c3-bf36-58c3e04cb026" containerName="revoke-keys" Nov 04 01:48:17.923529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.923528 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2bcbe1-d138-40c3-bf36-58c3e04cb026" containerName="revoke-keys" Nov 04 01:48:17.923612 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.923603 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2bcbe1-d138-40c3-bf36-58c3e04cb026" containerName="revoke-keys" Nov 04 01:48:17.926840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.926822 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:17.981886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:17.981854 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d"] Nov 04 01:48:18.024533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.024446 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.024716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.024606 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5n67\" (UniqueName: \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.024760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.024741 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.025376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.025358 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:48:18.025418 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.025364 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:48:18.125918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.125873 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.126172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.125933 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.126172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.126075 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5n67\" (UniqueName: \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.126715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.126692 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.128477 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.128448 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.182247 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.182175 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5n67\" (UniqueName: \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.240451 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.240401 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:18.616104 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:18.616068 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d"] Nov 04 01:48:18.617105 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:48:18.617075 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2bdcf_c56f_4c28_9472_4d7801dabd0e.slice/crio-9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48 WatchSource:0}: Error finding container 9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48: Status 404 returned error can't find the container with id 9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48 Nov 04 01:48:19.281199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:19.281104 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" event={"ID":"66c2bdcf-c56f-4c28-9472-4d7801dabd0e","Type":"ContainerStarted","Data":"be7b0777419dcfbaa38d84af9fdd15561cce346db36ec63e409da5c559236c46"} Nov 04 01:48:19.281199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:19.281160 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" event={"ID":"66c2bdcf-c56f-4c28-9472-4d7801dabd0e","Type":"ContainerStarted","Data":"9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48"} Nov 04 01:48:19.394888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:19.394824 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" podStartSLOduration=2.394809636 podStartE2EDuration="2.394809636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:19.393117487 +0000 UTC m=+3068.151779647" watchObservedRunningTime="2026-11-04 01:48:19.394809636 +0000 UTC m=+3068.153471796" Nov 04 01:48:20.288953 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:20.288914 4486 generic.go:356] "Generic (PLEG): container finished" podID="66c2bdcf-c56f-4c28-9472-4d7801dabd0e" containerID="be7b0777419dcfbaa38d84af9fdd15561cce346db36ec63e409da5c559236c46" exitCode=0 Nov 04 01:48:20.289453 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:20.289022 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" event={"ID":"66c2bdcf-c56f-4c28-9472-4d7801dabd0e","Type":"ContainerDied","Data":"be7b0777419dcfbaa38d84af9fdd15561cce346db36ec63e409da5c559236c46"} Nov 04 01:48:21.441012 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.440956 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:21.459715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.459677 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\") pod \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " Nov 04 01:48:21.459715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.459717 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\" (UniqueName: \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\") pod \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " Nov 04 01:48:21.459973 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.459780 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\") pod \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\" (UID: \"66c2bdcf-c56f-4c28-9472-4d7801dabd0e\") " Nov 04 01:48:21.460289 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.460094 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca" pod "66c2bdcf-c56f-4c28-9472-4d7801dabd0e" (UID: "66c2bdcf-c56f-4c28-9472-4d7801dabd0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:21.460402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.460296 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:21.462144 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.462120 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca" pod "66c2bdcf-c56f-4c28-9472-4d7801dabd0e" (UID: "66c2bdcf-c56f-4c28-9472-4d7801dabd0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:21.462404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.462387 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67" pod "66c2bdcf-c56f-4c28-9472-4d7801dabd0e" (UID: "66c2bdcf-c56f-4c28-9472-4d7801dabd0e"). InnerVolumeSpecName "kube-api-access-m5n67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:21.526436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.526395 4486 scope.go:122] "RemoveContainer" containerID="504cc52db8d34e19d05f0faa205b29e58a59accba2d878555ec3729d299e5dec" Nov 04 01:48:21.536698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.536674 4486 scope.go:122] "RemoveContainer" containerID="5e47e392b646180ba1f41677027f8ed0c29f203ae234991e68fdf8d6b77cbd9c" Nov 04 01:48:21.546093 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.546072 4486 scope.go:122] "RemoveContainer" containerID="2ed3af2836fc327c1665f5fab17d84c1feda766b4c56b005ce3969f8930e4e5f" Nov 04 01:48:21.563190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.562138 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5n67\" (UniqueName: \"kubernetes.io/projected/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-kube-api-access-m5n67\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:21.563190 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:21.562187 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c2bdcf-c56f-4c28-9472-4d7801dabd0e-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:22.302286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:22.302246 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" event={"ID":"66c2bdcf-c56f-4c28-9472-4d7801dabd0e","Type":"ContainerDied","Data":"9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48"} Nov 04 01:48:22.302286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:22.302290 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d" Nov 04 01:48:22.302581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:22.302341 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9686df6b501071d0e8558362371711cdeae754f1b3b5ab21c0fa059591717d48" Nov 04 01:48:22.310488 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:48:22.310435 4486 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2bdcf_c56f_4c28_9472_4d7801dabd0e.slice/cpuset.cpus.effective": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2bdcf_c56f_4c28_9472_4d7801dabd0e.slice/cpuset.cpus.effective: no such device Nov 04 01:48:22.438279 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:22.438242 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:48:34.532601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:34.532559 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d"] Nov 04 01:48:34.579846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:34.579803 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f4981c-trv5d"] Nov 04 01:48:35.348464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:35.348423 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c2bdcf-c56f-4c28-9472-4d7801dabd0e" path="/var/lib/kubelet/pods/66c2bdcf-c56f-4c28-9472-4d7801dabd0e/volumes" Nov 04 01:48:37.458440 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.458382 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" podUID="caf87389-7c4a-4a38-852a-afae769f51fc" containerName="main" containerID="cri-o://37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99" gracePeriod=45 Nov 04 01:48:37.725360 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.725296 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:48:37.836404 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836361 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.836683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836413 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.836683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836468 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.836683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836504 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.836683 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836629 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\" (UniqueName: \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.836951 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.836713 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\") pod \"caf87389-7c4a-4a38-852a-afae769f51fc\" (UID: \"caf87389-7c4a-4a38-852a-afae769f51fc\") " Nov 04 01:48:37.837391 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.837359 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:37.837952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.837634 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:37.837952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.837645 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:37.839232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.839191 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:37.839481 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.839456 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:37.841030 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.839956 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9" pod "caf87389-7c4a-4a38-852a-afae769f51fc" (UID: "caf87389-7c4a-4a38-852a-afae769f51fc"). InnerVolumeSpecName "kube-api-access-w4hj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:37.938374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938338 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hj9\" (UniqueName: \"kubernetes.io/projected/caf87389-7c4a-4a38-852a-afae769f51fc-kube-api-access-w4hj9\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:37.938374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938370 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:37.938374 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938381 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:37.938631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938391 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf87389-7c4a-4a38-852a-afae769f51fc-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:37.938631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938399 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:37.938631 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:37.938408 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf87389-7c4a-4a38-852a-afae769f51fc-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:38.409598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.409562 4486 generic.go:356] "Generic (PLEG): container finished" podID="caf87389-7c4a-4a38-852a-afae769f51fc" containerID="37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99" exitCode=0 Nov 04 01:48:38.409781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.409605 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" Nov 04 01:48:38.409781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.409644 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" event={"ID":"caf87389-7c4a-4a38-852a-afae769f51fc","Type":"ContainerDied","Data":"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99"} Nov 04 01:48:38.409781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.409684 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k" event={"ID":"caf87389-7c4a-4a38-852a-afae769f51fc","Type":"ContainerDied","Data":"8fd88b7c118186dba35bf3ce7bd26f3480ab87e69f2a24979dd66eacdf7624c7"} Nov 04 01:48:38.409781 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.409716 4486 scope.go:122] "RemoveContainer" containerID="37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99" Nov 04 01:48:38.420056 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.420033 4486 scope.go:122] "RemoveContainer" containerID="37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99" Nov 04 01:48:38.420359 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:48:38.420339 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99\": container with ID starting with 37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99 not found: ID does not exist" containerID="37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99" Nov 04 01:48:38.420401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.420367 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99"} err="failed to get container status \"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99\": rpc error: code = NotFound desc = could not find container \"37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99\": container with ID starting with 37649a6bf507ec6b5c2f41306470ce514c6fc264a057bac934c3e36cf9adfd99 not found: ID does not exist" Nov 04 01:48:38.566493 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.566455 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:48:38.613760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:38.613723 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f4981c/simulator-single-main-f4981c-kserve-775595c45-kk54k"] Nov 04 01:48:39.347735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:39.347696 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf87389-7c4a-4a38-852a-afae769f51fc" path="/var/lib/kubelet/pods/caf87389-7c4a-4a38-852a-afae769f51fc/volumes" Nov 04 01:48:41.804235 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:41.804198 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:48:41.804699 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:41.804423 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" containerID="cri-o://7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed" gracePeriod=30 Nov 04 01:48:43.831638 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:48:43.831600 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:48:44.048495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:44.048450 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh"] Nov 04 01:48:44.095590 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:44.095496 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2811e627-znfmh"] Nov 04 01:48:45.348829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:45.348794 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142a5a5a-0a63-4e06-aec9-8cb86b8a5e06" path="/var/lib/kubelet/pods/142a5a5a-0a63-4e06-aec9-8cb86b8a5e06/volumes" Nov 04 01:48:47.053530 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.053502 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:48:47.144935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.144830 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.144935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.144884 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.144951 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145011 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145057 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145094 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145142 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145191 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.145223 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145203 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:47.145532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145357 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:47.145594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145536 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.145594 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145558 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.145698 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145658 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:47.145776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.145754 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:47.147400 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.147366 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "kube-api-access-f55qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:47.147548 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.147532 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:47.147812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.147794 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:48:47.147867 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.147799 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:47.246237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246201 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\") pod \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\" (UID: \"d138dd76-3ddd-4ed3-ada5-2d1f164321a1\") " Nov 04 01:48:47.246536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246519 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246541 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246553 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55qs\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-kube-api-access-f55qs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246565 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246574 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246583 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.246709 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.246576 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert" pod "d138dd76-3ddd-4ed3-ada5-2d1f164321a1" (UID: "d138dd76-3ddd-4ed3-ada5-2d1f164321a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:48:47.348266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.348235 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d138dd76-3ddd-4ed3-ada5-2d1f164321a1-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:47.468715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.468612 4486 generic.go:356] "Generic (PLEG): container finished" podID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerID="7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed" exitCode=0 Nov 04 01:48:47.468715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.468674 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" Nov 04 01:48:47.468715 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.468677 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerDied","Data":"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed"} Nov 04 01:48:47.469028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.468762 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst" event={"ID":"d138dd76-3ddd-4ed3-ada5-2d1f164321a1","Type":"ContainerDied","Data":"97747e552da196dbf349d03d8a05182b0ca2f2498272f9a8243a738493e3b8e1"} Nov 04 01:48:47.469028 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.468789 4486 scope.go:122] "RemoveContainer" containerID="7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed" Nov 04 01:48:47.479075 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.478969 4486 scope.go:122] "RemoveContainer" containerID="e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607" Nov 04 01:48:47.489469 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.489448 4486 scope.go:122] "RemoveContainer" containerID="7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed" Nov 04 01:48:47.489850 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:48:47.489827 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed\": container with ID starting with 7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed not found: ID does not exist" containerID="7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed" Nov 04 01:48:47.489905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.489863 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed"} err="failed to get container status \"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed\": rpc error: code = NotFound desc = could not find container \"7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed\": container with ID starting with 7abe3c57c742ebbea652dcd9119e75218e3917cc414e1db77a71177b8504a9ed not found: ID does not exist" Nov 04 01:48:47.489905 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.489882 4486 scope.go:122] "RemoveContainer" containerID="e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607" Nov 04 01:48:47.490212 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:48:47.490193 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607\": container with ID starting with e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607 not found: ID does not exist" containerID="e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607" Nov 04 01:48:47.490263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.490220 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607"} err="failed to get container status \"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607\": rpc error: code = NotFound desc = could not find container \"e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607\": container with ID starting with e72923042b08bcadb452f172045ae6cfdec3a5a48271e5aadaf4a04d3f0a0607 not found: ID does not exist" Nov 04 01:48:47.629291 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.629254 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:48:47.634409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:47.634382 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f4981c-openshift-default-66d84cf56fzhwst"] Nov 04 01:48:48.874572 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.874526 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.874958 4486 cpu_manager.go:414] "containerMap: removing container" podUID="caf87389-7c4a-4a38-852a-afae769f51fc" containerName="main" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.874969 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf87389-7c4a-4a38-852a-afae769f51fc" containerName="main" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875003 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875009 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875018 4486 cpu_manager.go:414] "containerMap: removing container" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875023 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875042 4486 cpu_manager.go:414] "containerMap: removing container" podUID="66c2bdcf-c56f-4c28-9472-4d7801dabd0e" containerName="revoke-keys" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875047 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c2bdcf-c56f-4c28-9472-4d7801dabd0e" containerName="revoke-keys" Nov 04 01:48:48.875110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875109 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c2bdcf-c56f-4c28-9472-4d7801dabd0e" containerName="revoke-keys" Nov 04 01:48:48.875401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875120 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf87389-7c4a-4a38-852a-afae769f51fc" containerName="main" Nov 04 01:48:48.875401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875128 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.875401 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.875134 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" containerName="istio-proxy" Nov 04 01:48:48.879711 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.879690 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.934946 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.934910 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:48:48.964325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964290 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.964325 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964329 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.964564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964409 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.964564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964484 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.964564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964535 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99wj\" (UniqueName: \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.964684 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.964646 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:48.977513 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:48.977483 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4889a1b9-kserve-self-signed-certs\"" Nov 04 01:48:49.065276 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065236 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065287 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065308 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065342 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99wj\" (UniqueName: \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065429 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065458 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065734 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065748 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.065829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.065798 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.067746 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.067721 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.067922 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.067904 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.120593 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.120553 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99wj\" (UniqueName: \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\") pod \"e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.192775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.192686 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:49.347760 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.347719 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d138dd76-3ddd-4ed3-ada5-2d1f164321a1" path="/var/lib/kubelet/pods/d138dd76-3ddd-4ed3-ada5-2d1f164321a1/volumes" Nov 04 01:48:49.565149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:49.565120 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:48:49.566945 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:48:49.566918 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c8ea7a8_ae1e_498e_98e6_40b062374a19.slice/crio-d832d1d434aea58db1b5fe013d1714e7b3ca80b08646ff4d672464eb8a6e904a WatchSource:0}: Error finding container d832d1d434aea58db1b5fe013d1714e7b3ca80b08646ff4d672464eb8a6e904a: Status 404 returned error can't find the container with id d832d1d434aea58db1b5fe013d1714e7b3ca80b08646ff4d672464eb8a6e904a Nov 04 01:48:50.489893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:50.489845 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" event={"ID":"1c8ea7a8-ae1e-498e-98e6-40b062374a19","Type":"ContainerStarted","Data":"482eb18855d2b96138b9ac0b8ce6c76a234050d9a4ba5dc27b580f1325d6874b"} Nov 04 01:48:50.489893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:50.489895 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" event={"ID":"1c8ea7a8-ae1e-498e-98e6-40b062374a19","Type":"ContainerStarted","Data":"d832d1d434aea58db1b5fe013d1714e7b3ca80b08646ff4d672464eb8a6e904a"} Nov 04 01:48:50.601249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:50.601191 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" podStartSLOduration=2.601173828 podStartE2EDuration="2.601173828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:50.599016126 +0000 UTC m=+3099.357678288" watchObservedRunningTime="2026-11-04 01:48:50.601173828 +0000 UTC m=+3099.359835990" Nov 04 01:48:51.193185 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:51.193148 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:59.193801 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:59.193709 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:59.206895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:59.206866 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:48:59.560629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:48:59.560546 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:49:13.831619 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:49:13.831576 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:49:22.046353 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:22.046240 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh"] Nov 04 01:49:22.093380 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:22.093342 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pl4zh"] Nov 04 01:49:23.348181 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:23.348146 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65963ab9-52fd-4ad8-820f-bb1672596211" path="/var/lib/kubelet/pods/65963ab9-52fd-4ad8-820f-bb1672596211/volumes" Nov 04 01:49:33.409532 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.409491 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:49:33.414601 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.414578 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.512297 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.512257 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4889a1b9-kserve-self-signed-certs\"" Nov 04 01:49:33.515197 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.515165 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:49:33.562246 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562210 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.562438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562265 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5297\" (UniqueName: \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.562438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562355 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.562438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562385 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.562618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562554 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.562663 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.562636 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664139 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664198 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5297\" (UniqueName: \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664240 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664271 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664332 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664367 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664570 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664661 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.664755 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.664680 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.666592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.666568 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.666901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.666882 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.719597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.719559 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5297\" (UniqueName: \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\") pod \"e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:33.727445 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:33.727417 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:34.106438 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:34.106407 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:49:34.107691 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:49:34.107661 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee31418a_ae71_4c0a_870c_2e71a4f1b7f7.slice/crio-a21aed5d6d54bb1a554958a01c38ba8db95e541bae7853afd9c8533258810e06 WatchSource:0}: Error finding container a21aed5d6d54bb1a554958a01c38ba8db95e541bae7853afd9c8533258810e06: Status 404 returned error can't find the container with id a21aed5d6d54bb1a554958a01c38ba8db95e541bae7853afd9c8533258810e06 Nov 04 01:49:34.759775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:34.759738 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" event={"ID":"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7","Type":"ContainerStarted","Data":"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3"} Nov 04 01:49:34.759775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:34.759781 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" event={"ID":"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7","Type":"ContainerStarted","Data":"a21aed5d6d54bb1a554958a01c38ba8db95e541bae7853afd9c8533258810e06"} Nov 04 01:49:34.872270 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:34.872211 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" podStartSLOduration=1.872196357 podStartE2EDuration="1.872196357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:34.869617687 +0000 UTC m=+3143.628279847" watchObservedRunningTime="2026-11-04 01:49:34.872196357 +0000 UTC m=+3143.630858563" Nov 04 01:49:35.728403 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:35.728367 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:43.728039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:43.727958 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:43.741068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:43.741036 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:43.834227 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:49:43.834189 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:49:43.846818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:43.846779 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:49:55.173900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.173858 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r"] Nov 04 01:49:55.177772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.177755 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.231810 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.231755 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r"] Nov 04 01:49:55.244119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.244085 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qfq\" (UniqueName: \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.244316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.244207 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.244316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.244289 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.275342 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.275311 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:49:55.275518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.275456 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:49:55.344886 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.344834 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qfq\" (UniqueName: \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.345138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.344916 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.345138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.344979 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.345700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.345673 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.347838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.347811 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.399649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.399608 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qfq\" (UniqueName: \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\") pod \"maas-api-revoke-sub-452b6d7b-t6l9r\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.491874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.491784 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:55.868500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.868472 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r"] Nov 04 01:49:55.871141 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:49:55.871109 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84331de9_69c6_4e96_88ed_ae88ec782034.slice/crio-6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0 WatchSource:0}: Error finding container 6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0: Status 404 returned error can't find the container with id 6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0 Nov 04 01:49:55.910794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:55.910760 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" event={"ID":"84331de9-69c6-4e96-88ed-ae88ec782034","Type":"ContainerStarted","Data":"6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0"} Nov 04 01:49:56.918629 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:56.918595 4486 generic.go:356] "Generic (PLEG): container finished" podID="84331de9-69c6-4e96-88ed-ae88ec782034" containerID="137c351879b8f2f884d90305b5a2fe1351051d77b38ab84a1f818565db7eab56" exitCode=0 Nov 04 01:49:56.919061 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:56.918638 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" event={"ID":"84331de9-69c6-4e96-88ed-ae88ec782034","Type":"ContainerDied","Data":"137c351879b8f2f884d90305b5a2fe1351051d77b38ab84a1f818565db7eab56"} Nov 04 01:49:58.100925 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.100888 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:58.179972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.179870 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\" (UniqueName: \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\") pod \"84331de9-69c6-4e96-88ed-ae88ec782034\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " Nov 04 01:49:58.179972 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.179940 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\") pod \"84331de9-69c6-4e96-88ed-ae88ec782034\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " Nov 04 01:49:58.180249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.180039 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\") pod \"84331de9-69c6-4e96-88ed-ae88ec782034\" (UID: \"84331de9-69c6-4e96-88ed-ae88ec782034\") " Nov 04 01:49:58.182275 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.182243 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq" pod "84331de9-69c6-4e96-88ed-ae88ec782034" (UID: "84331de9-69c6-4e96-88ed-ae88ec782034"). InnerVolumeSpecName "kube-api-access-b2qfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:58.194833 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.194800 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca" pod "84331de9-69c6-4e96-88ed-ae88ec782034" (UID: "84331de9-69c6-4e96-88ed-ae88ec782034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:58.196791 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.196751 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca" pod "84331de9-69c6-4e96-88ed-ae88ec782034" (UID: "84331de9-69c6-4e96-88ed-ae88ec782034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:58.281763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.281706 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qfq\" (UniqueName: \"kubernetes.io/projected/84331de9-69c6-4e96-88ed-ae88ec782034-kube-api-access-b2qfq\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:58.281763 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.281757 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:58.282091 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:58.281774 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84331de9-69c6-4e96-88ed-ae88ec782034-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:59.008850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:59.008801 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" event={"ID":"84331de9-69c6-4e96-88ed-ae88ec782034","Type":"ContainerDied","Data":"6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0"} Nov 04 01:49:59.014035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:59.009234 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r" Nov 04 01:49:59.014035 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:49:59.012584 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6077fbfc66d73e605feb0dfb45dea6fc2c22e790a4fd25efef14dace8026b0e0" Nov 04 01:50:08.017088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:08.017037 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:50:13.835120 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:50:13.835071 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:50:21.610053 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:21.610020 4486 scope.go:122] "RemoveContainer" containerID="65415ebab2ad1545f4d612fa64e609701b9935a77e07b6de6102ee7778486a13" Nov 04 01:50:21.620416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:21.620394 4486 scope.go:122] "RemoveContainer" containerID="52de49655507e8ee6e25a9cc538b25ddc743a6048d843e8f03728ddef79332ae" Nov 04 01:50:21.630478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:21.630455 4486 scope.go:122] "RemoveContainer" containerID="2b9ab38219443664083b2114034119b776d8248ecbf1428839d4ca775aadde7f" Nov 04 01:50:23.037701 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.037591 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" podUID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" containerName="main" containerID="cri-o://482eb18855d2b96138b9ac0b8ce6c76a234050d9a4ba5dc27b580f1325d6874b" gracePeriod=45 Nov 04 01:50:23.166219 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.166185 4486 generic.go:356] "Generic (PLEG): container finished" podID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" containerID="482eb18855d2b96138b9ac0b8ce6c76a234050d9a4ba5dc27b580f1325d6874b" exitCode=0 Nov 04 01:50:23.166394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.166240 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" event={"ID":"1c8ea7a8-ae1e-498e-98e6-40b062374a19","Type":"ContainerDied","Data":"482eb18855d2b96138b9ac0b8ce6c76a234050d9a4ba5dc27b580f1325d6874b"} Nov 04 01:50:23.296926 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.296858 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:50:23.396868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.396832 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.397132 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397099 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:23.397280 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397154 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.397357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397305 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:23.397357 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397315 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.397479 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397426 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.397543 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397477 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\" (UniqueName: \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.397607 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397558 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\") pod \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\" (UID: \"1c8ea7a8-ae1e-498e-98e6-40b062374a19\") " Nov 04 01:50:23.398184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397956 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:23.398184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.397997 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:23.398184 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.398146 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:23.409377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.409192 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:50:23.409377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.409334 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:23.409377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.409340 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj" pod "1c8ea7a8-ae1e-498e-98e6-40b062374a19" (UID: "1c8ea7a8-ae1e-498e-98e6-40b062374a19"). InnerVolumeSpecName "kube-api-access-c99wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:50:23.412405 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.411777 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:50:23.499466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.499428 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:23.499466 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.499462 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99wj\" (UniqueName: \"kubernetes.io/projected/1c8ea7a8-ae1e-498e-98e6-40b062374a19-kube-api-access-c99wj\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:23.499703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.499492 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:23.499703 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:23.499502 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c8ea7a8-ae1e-498e-98e6-40b062374a19-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:24.175749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:24.175716 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" Nov 04 01:50:24.176273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:24.175715 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h" event={"ID":"1c8ea7a8-ae1e-498e-98e6-40b062374a19","Type":"ContainerDied","Data":"d832d1d434aea58db1b5fe013d1714e7b3ca80b08646ff4d672464eb8a6e904a"} Nov 04 01:50:24.176273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:24.175848 4486 scope.go:122] "RemoveContainer" containerID="482eb18855d2b96138b9ac0b8ce6c76a234050d9a4ba5dc27b580f1325d6874b" Nov 04 01:50:24.333471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:24.333435 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:50:24.380772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:24.380734 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4889a1b9-kserve-7695fcb8f-tcw6h"] Nov 04 01:50:25.348437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:25.348402 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" path="/var/lib/kubelet/pods/1c8ea7a8-ae1e-498e-98e6-40b062374a19/volumes" Nov 04 01:50:27.567829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.567796 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q"] Nov 04 01:50:27.568341 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568326 4486 cpu_manager.go:414] "containerMap: removing container" podUID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" containerName="main" Nov 04 01:50:27.568388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568342 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" containerName="main" Nov 04 01:50:27.568388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568355 4486 cpu_manager.go:414] "containerMap: removing container" podUID="84331de9-69c6-4e96-88ed-ae88ec782034" containerName="revoke-keys" Nov 04 01:50:27.568388 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568360 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84331de9-69c6-4e96-88ed-ae88ec782034" containerName="revoke-keys" Nov 04 01:50:27.568489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568432 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8ea7a8-ae1e-498e-98e6-40b062374a19" containerName="main" Nov 04 01:50:27.568489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.568440 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="84331de9-69c6-4e96-88ed-ae88ec782034" containerName="revoke-keys" Nov 04 01:50:27.573286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.573254 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.623686 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.623650 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:50:27.623860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.623802 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:50:27.626111 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.626076 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q"] Nov 04 01:50:27.649458 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.649423 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.649654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.649510 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.649654 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.649553 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg44p\" (UniqueName: \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.750776 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.750735 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg44p\" (UniqueName: \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.750974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.750848 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.750974 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.750888 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.751733 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.751702 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.753420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.753393 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.806546 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.806493 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg44p\" (UniqueName: \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\") pod \"maas-api-revoke-sub-3e31a506-whq2q\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:27.890599 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:27.890512 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:28.266172 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:28.266127 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q"] Nov 04 01:50:28.267469 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:50:28.267443 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab7397f_f5c5_40e0_a5bd_13268a15a0ba.slice/crio-94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f WatchSource:0}: Error finding container 94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f: Status 404 returned error can't find the container with id 94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f Nov 04 01:50:29.210727 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:29.210686 4486 generic.go:356] "Generic (PLEG): container finished" podID="5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" containerID="b0c8d0d22465cd0075763c917c0f3151abb9a935c1357c5c4e18b0a91ce19bfa" exitCode=0 Nov 04 01:50:29.211201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:29.210776 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" event={"ID":"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba","Type":"ContainerDied","Data":"b0c8d0d22465cd0075763c917c0f3151abb9a935c1357c5c4e18b0a91ce19bfa"} Nov 04 01:50:29.211201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:29.210844 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" event={"ID":"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba","Type":"ContainerStarted","Data":"94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f"} Nov 04 01:50:30.667008 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.664364 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:30.822193 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.822160 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\") pod \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " Nov 04 01:50:30.822384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.822218 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\") pod \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " Nov 04 01:50:30.822384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.822291 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\" (UniqueName: \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\") pod \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\" (UID: \"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba\") " Nov 04 01:50:30.823042 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.823014 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca" pod "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" (UID: "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:50:30.824752 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.824723 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca" pod "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" (UID: "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:50:30.825127 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.825100 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p" pod "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" (UID: "5ab7397f-f5c5-40e0-a5bd-13268a15a0ba"). InnerVolumeSpecName "kube-api-access-tg44p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:50:30.925838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.924733 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:30.925838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.924771 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:30.925838 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:30.924786 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg44p\" (UniqueName: \"kubernetes.io/projected/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba-kube-api-access-tg44p\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:31.226029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:31.225904 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" Nov 04 01:50:31.226029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:31.225915 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q" event={"ID":"5ab7397f-f5c5-40e0-a5bd-13268a15a0ba","Type":"ContainerDied","Data":"94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f"} Nov 04 01:50:31.226029 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:31.225972 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f970435548887284dc1a35166a7a64608e620093b126cfa95503a24416893f" Nov 04 01:50:38.438707 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.438658 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" podUID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" containerName="main" containerID="cri-o://a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3" gracePeriod=45 Nov 04 01:50:38.691856 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.691795 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:50:38.809742 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.809693 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.809941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.809860 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.809941 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.809923 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.810110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.810021 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.810110 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.810065 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.810217 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.810135 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\" (UniqueName: \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\") pod \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\" (UID: \"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7\") " Nov 04 01:50:38.813578 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.813541 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "kube-api-access-x5297". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:50:38.814254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.814217 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:38.814409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.814366 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:38.816117 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.815788 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:38.817693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.817666 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:50:38.824637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.824609 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm" pod "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" (UID: "ee31418a-ae71-4c0a-870c-2e71a4f1b7f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:50:38.911476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911442 4486 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-dshm\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:38.911476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911473 4486 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tmp-dir\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:38.911695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911486 4486 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-model-cache\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:38.911695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911499 4486 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-home\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:38.911695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911514 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5297\" (UniqueName: \"kubernetes.io/projected/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-kube-api-access-x5297\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:38.911695 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:38.911527 4486 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7-tls-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:50:39.276585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.276541 4486 generic.go:356] "Generic (PLEG): container finished" podID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" containerID="a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3" exitCode=0 Nov 04 01:50:39.276775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.276599 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" Nov 04 01:50:39.276775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.276629 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" event={"ID":"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7","Type":"ContainerDied","Data":"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3"} Nov 04 01:50:39.276775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.276675 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x" event={"ID":"ee31418a-ae71-4c0a-870c-2e71a4f1b7f7","Type":"ContainerDied","Data":"a21aed5d6d54bb1a554958a01c38ba8db95e541bae7853afd9c8533258810e06"} Nov 04 01:50:39.276775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.276695 4486 scope.go:122] "RemoveContainer" containerID="a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3" Nov 04 01:50:39.287119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.287093 4486 scope.go:122] "RemoveContainer" containerID="a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3" Nov 04 01:50:39.287458 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:50:39.287436 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3\": container with ID starting with a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3 not found: ID does not exist" containerID="a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3" Nov 04 01:50:39.287518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.287470 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3"} err="failed to get container status \"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3\": rpc error: code = NotFound desc = could not find container \"a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3\": container with ID starting with a545f500240ce0bcae4249294d2a0fd24e7918f55316a45d0fffbfe9f1acdbe3 not found: ID does not exist" Nov 04 01:50:39.435309 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.435267 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:50:39.482667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:39.482631 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4889a1b9-kserve-6774fd8fc-rw44x"] Nov 04 01:50:41.389034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:50:41.388327 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" path="/var/lib/kubelet/pods/ee31418a-ae71-4c0a-870c-2e71a4f1b7f7/volumes" Nov 04 01:50:43.832161 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:50:43.832126 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:51:00.266346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266307 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm"] Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266748 4486 cpu_manager.go:414] "containerMap: removing container" podUID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" containerName="main" Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266760 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" containerName="main" Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266775 4486 cpu_manager.go:414] "containerMap: removing container" podUID="5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" containerName="revoke-keys" Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266780 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" containerName="revoke-keys" Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266862 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee31418a-ae71-4c0a-870c-2e71a4f1b7f7" containerName="main" Nov 04 01:51:00.267130 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.266871 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" containerName="revoke-keys" Nov 04 01:51:00.270382 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.270358 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.294564 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.294535 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.294735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.294587 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.294735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.294612 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9pp\" (UniqueName: \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.324869 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.324839 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm"] Nov 04 01:51:00.367997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.367950 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qb9zm\"" Nov 04 01:51:00.367997 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.367950 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:51:00.396097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.396059 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.396331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.396161 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.396331 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.396203 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9pp\" (UniqueName: \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.396918 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.396887 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.398542 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.398511 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.450965 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.450929 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9pp\" (UniqueName: \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\") pod \"maas-api-revoke-sub-7ca319fb-9c6jm\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.588757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.588717 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:00.959258 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:00.959223 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm"] Nov 04 01:51:00.959420 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:51:00.959231 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a653897_36f7_47f0_94eb_858b50a55dc7.slice/crio-8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5 WatchSource:0}: Error finding container 8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5: Status 404 returned error can't find the container with id 8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5 Nov 04 01:51:01.456361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:01.456317 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" event={"ID":"3a653897-36f7-47f0-94eb-858b50a55dc7","Type":"ContainerStarted","Data":"071be81f5618d82e2b94cc5a7d99f1b2febd5752472a0e0ec3a0f24863eb13b4"} Nov 04 01:51:01.456893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:01.456391 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" event={"ID":"3a653897-36f7-47f0-94eb-858b50a55dc7","Type":"ContainerStarted","Data":"8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5"} Nov 04 01:51:01.565464 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:01.565403 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" podStartSLOduration=1.565388851 podStartE2EDuration="1.565388851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:01.563713327 +0000 UTC m=+3230.322375488" watchObservedRunningTime="2026-11-04 01:51:01.565388851 +0000 UTC m=+3230.324051012" Nov 04 01:51:02.464298 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:02.464256 4486 generic.go:356] "Generic (PLEG): container finished" podID="3a653897-36f7-47f0-94eb-858b50a55dc7" containerID="071be81f5618d82e2b94cc5a7d99f1b2febd5752472a0e0ec3a0f24863eb13b4" exitCode=0 Nov 04 01:51:02.464734 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:02.464346 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" event={"ID":"3a653897-36f7-47f0-94eb-858b50a55dc7","Type":"ContainerDied","Data":"071be81f5618d82e2b94cc5a7d99f1b2febd5752472a0e0ec3a0f24863eb13b4"} Nov 04 01:51:03.626268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.626242 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:03.727580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.727488 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\") pod \"3a653897-36f7-47f0-94eb-858b50a55dc7\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " Nov 04 01:51:03.727580 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.727563 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\") pod \"3a653897-36f7-47f0-94eb-858b50a55dc7\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " Nov 04 01:51:03.727792 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.727583 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\" (UniqueName: \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\") pod \"3a653897-36f7-47f0-94eb-858b50a55dc7\" (UID: \"3a653897-36f7-47f0-94eb-858b50a55dc7\") " Nov 04 01:51:03.727969 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.727938 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca" pod "3a653897-36f7-47f0-94eb-858b50a55dc7" (UID: "3a653897-36f7-47f0-94eb-858b50a55dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:03.729754 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.729724 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp" pod "3a653897-36f7-47f0-94eb-858b50a55dc7" (UID: "3a653897-36f7-47f0-94eb-858b50a55dc7"). InnerVolumeSpecName "kube-api-access-dx9pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:03.729893 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.729784 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca" pod "3a653897-36f7-47f0-94eb-858b50a55dc7" (UID: "3a653897-36f7-47f0-94eb-858b50a55dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:03.829230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.829187 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:03.829230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.829219 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a653897-36f7-47f0-94eb-858b50a55dc7-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:03.829230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:03.829229 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx9pp\" (UniqueName: \"kubernetes.io/projected/3a653897-36f7-47f0-94eb-858b50a55dc7-kube-api-access-dx9pp\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:04.476888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:04.476851 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" Nov 04 01:51:04.476888 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:04.476866 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm" event={"ID":"3a653897-36f7-47f0-94eb-858b50a55dc7","Type":"ContainerDied","Data":"8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5"} Nov 04 01:51:04.477142 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:04.476909 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8381ccfce9624a4ac98b527587fcbc8b4ddc2aeda8cdd4a6c13af5b8cf9826c5" Nov 04 01:51:11.045384 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:11.045343 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d"] Nov 04 01:51:11.092681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:11.092637 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7rv4d"] Nov 04 01:51:11.348090 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:11.348046 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df29f09-f5c9-419c-98cc-4c73f7df2c95" path="/var/lib/kubelet/pods/1df29f09-f5c9-419c-98cc-4c73f7df2c95/volumes" Nov 04 01:51:13.831740 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:51:13.831696 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:51:20.604305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.604265 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:51:20.604726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.604712 4486 cpu_manager.go:414] "containerMap: removing container" podUID="3a653897-36f7-47f0-94eb-858b50a55dc7" containerName="revoke-keys" Nov 04 01:51:20.604726 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.604723 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a653897-36f7-47f0-94eb-858b50a55dc7" containerName="revoke-keys" Nov 04 01:51:20.604817 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.604806 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a653897-36f7-47f0-94eb-858b50a55dc7" containerName="revoke-keys" Nov 04 01:51:20.608415 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.608395 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.703510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.703471 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:51:20.706821 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.706787 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.706830 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.706866 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwl4\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707021 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.706973 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.707032 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.707066 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.707136 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707212 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.707173 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.707412 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.707220 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.746866 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.746829 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4955ec16-openshift-default-dockercfg-jt46m\"" Nov 04 01:51:20.808484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808439 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808485 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808521 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwl4\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808554 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808577 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808608 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808666 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808699 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.808764 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808748 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.809201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.808943 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.809201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.809136 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.809201 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.809138 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.809372 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.809278 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.809512 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.809490 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.811114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.811092 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.811216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.811161 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.863496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.863423 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.863659 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.863509 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwl4\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\") pod \"e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:20.921827 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:20.921784 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:21.350637 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.350610 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:51:21.354317 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:51:21.354292 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260dabb7_90dc_4fd0_9792_6cc0dccf1be9.slice/crio-8b2f4561dd49816cf18d6eb84d0de07047ef7894deda11673dbe1e499d8b95f5 WatchSource:0}: Error finding container 8b2f4561dd49816cf18d6eb84d0de07047ef7894deda11673dbe1e499d8b95f5: Status 404 returned error can't find the container with id 8b2f4561dd49816cf18d6eb84d0de07047ef7894deda11673dbe1e499d8b95f5 Nov 04 01:51:21.356850 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.356806 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:21.357034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.356930 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:21.357034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.356997 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:21.584431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.584386 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerStarted","Data":"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6"} Nov 04 01:51:21.584646 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.584459 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerStarted","Data":"8b2f4561dd49816cf18d6eb84d0de07047ef7894deda11673dbe1e499d8b95f5"} Nov 04 01:51:21.698968 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.698903 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podStartSLOduration=1.6988824949999999 podStartE2EDuration="1.698882495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:21.695794046 +0000 UTC m=+3250.454456209" watchObservedRunningTime="2026-11-04 01:51:21.698882495 +0000 UTC m=+3250.457544657" Nov 04 01:51:21.922520 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:21.922468 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:22.927878 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:22.927843 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:23.599474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:23.599432 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:23.600508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:23.600477 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:39.056014 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:39.055955 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5"] Nov 04 01:51:39.099313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:39.099279 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ld8f5"] Nov 04 01:51:39.359208 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:39.359173 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="350fb998-9caf-4c45-9d9d-aac988c53bb1" path="/var/lib/kubelet/pods/350fb998-9caf-4c45-9d9d-aac988c53bb1/volumes" Nov 04 01:51:43.833033 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:51:43.832975 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:51:45.739048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.738933 4486 generic.go:356] "Generic (PLEG): container finished" podID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerID="a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6" exitCode=0 Nov 04 01:51:45.739048 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.739013 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerDied","Data":"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6"} Nov 04 01:51:45.739510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.739470 4486 scope.go:122] "RemoveContainer" containerID="a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6" Nov 04 01:51:45.740775 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.740744 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:45.740832 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.740818 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:45.740870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:45.740854 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:51:46.762097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:46.762063 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerStarted","Data":"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2"} Nov 04 01:51:46.922320 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:46.922290 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:46.922518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:46.922499 4486 patch_prober.go:29] interesting pod/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 04 01:51:46.922574 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:46.922557 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 04 01:51:47.922849 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:47.922812 4486 patch_prober.go:29] interesting pod/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 04 01:51:47.923295 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:47.922873 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 04 01:51:48.922196 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:48.922162 4486 patch_prober.go:29] interesting pod/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 04 01:51:48.922375 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:48.922230 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 04 01:51:49.922108 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:49.922072 4486 patch_prober.go:29] interesting pod/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 04 01:51:49.922529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:49.922157 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 04 01:51:50.922376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:50.922335 4486 patch_prober.go:29] interesting pod/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Nov 04 01:51:50.922813 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:50.922395 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Nov 04 01:51:51.927114 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:51.927073 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:51.927537 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:51.927359 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:51.928214 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:51.928154 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:51:55.176992 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.176896 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz"] Nov 04 01:51:55.180790 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.180768 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.230667 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.230625 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.230868 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.230769 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8nv\" (UniqueName: \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.230952 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.230898 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.234921 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.234890 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz"] Nov 04 01:51:55.279066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.279031 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:51:55.279066 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.279064 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pm2xk\"" Nov 04 01:51:55.331859 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.331821 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8nv\" (UniqueName: \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.332087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.331901 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.332087 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.332037 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.332713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.332681 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.334379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.334348 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.386581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.386539 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8nv\" (UniqueName: \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\") pod \"maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.494592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.494496 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:55.865640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:55.865605 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz"] Nov 04 01:51:55.866886 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:51:55.866859 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023b020c_a522_44fb_b1a7_370e388ca42d.slice/crio-9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d WatchSource:0}: Error finding container 9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d: Status 404 returned error can't find the container with id 9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d Nov 04 01:51:56.828064 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:56.828025 4486 generic.go:356] "Generic (PLEG): container finished" podID="023b020c-a522-44fb-b1a7-370e388ca42d" containerID="dd0ffd9eb00ec9b05f64391c35562c53cf8ea7dcb4234e2da5f747c2763e7325" exitCode=0 Nov 04 01:51:56.828505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:56.828117 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" event={"ID":"023b020c-a522-44fb-b1a7-370e388ca42d","Type":"ContainerDied","Data":"dd0ffd9eb00ec9b05f64391c35562c53cf8ea7dcb4234e2da5f747c2763e7325"} Nov 04 01:51:56.828505 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:56.828181 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" event={"ID":"023b020c-a522-44fb-b1a7-370e388ca42d","Type":"ContainerStarted","Data":"9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d"} Nov 04 01:51:57.976397 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:57.976360 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:58.056492 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.056460 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\") pod \"023b020c-a522-44fb-b1a7-370e388ca42d\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " Nov 04 01:51:58.056655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.056623 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\" (UniqueName: \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\") pod \"023b020c-a522-44fb-b1a7-370e388ca42d\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " Nov 04 01:51:58.056655 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.056647 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\") pod \"023b020c-a522-44fb-b1a7-370e388ca42d\" (UID: \"023b020c-a522-44fb-b1a7-370e388ca42d\") " Nov 04 01:51:58.057131 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.057085 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca" pod "023b020c-a522-44fb-b1a7-370e388ca42d" (UID: "023b020c-a522-44fb-b1a7-370e388ca42d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:58.058619 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.058591 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca" pod "023b020c-a522-44fb-b1a7-370e388ca42d" (UID: "023b020c-a522-44fb-b1a7-370e388ca42d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:58.058745 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.058728 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv" pod "023b020c-a522-44fb-b1a7-370e388ca42d" (UID: "023b020c-a522-44fb-b1a7-370e388ca42d"). InnerVolumeSpecName "kube-api-access-9f8nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:58.158292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.158204 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8nv\" (UniqueName: \"kubernetes.io/projected/023b020c-a522-44fb-b1a7-370e388ca42d-kube-api-access-9f8nv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:58.158292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.158235 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:58.158292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.158245 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/023b020c-a522-44fb-b1a7-370e388ca42d-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:58.841186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.841150 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" Nov 04 01:51:58.841186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.841173 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz" event={"ID":"023b020c-a522-44fb-b1a7-370e388ca42d","Type":"ContainerDied","Data":"9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d"} Nov 04 01:51:58.841432 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:51:58.841243 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bef0f7211893ff356f0ad25eba1882deaf0c6b5f28e205d45dcc379891fec5d" Nov 04 01:52:13.833200 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:52:13.833158 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:52:14.689976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:14.689943 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz"] Nov 04 01:52:14.736875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:14.736834 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4955ec16-dr2hz"] Nov 04 01:52:15.348757 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:15.348722 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023b020c-a522-44fb-b1a7-370e388ca42d" path="/var/lib/kubelet/pods/023b020c-a522-44fb-b1a7-370e388ca42d/volumes" Nov 04 01:52:16.300910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:16.300869 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:52:16.301186 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:16.301164 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" containerID="cri-o://b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2" gracePeriod=30 Nov 04 01:52:18.111519 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.111479 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:52:18.111964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.111950 4486 cpu_manager.go:414] "containerMap: removing container" podUID="023b020c-a522-44fb-b1a7-370e388ca42d" containerName="revoke-keys" Nov 04 01:52:18.112051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.111965 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023b020c-a522-44fb-b1a7-370e388ca42d" containerName="revoke-keys" Nov 04 01:52:18.112103 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.112085 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="023b020c-a522-44fb-b1a7-370e388ca42d" containerName="revoke-keys" Nov 04 01:52:18.115313 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.115293 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.188476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.188443 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:52:18.208912 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.208878 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:52:18.211230 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.211205 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:52:18.215402 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.215373 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:52:18.234560 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.234532 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:52:18.258417 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.258389 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-3e5e0c-openshift-default-dockercfg-r9tkb\"" Nov 04 01:52:18.294164 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294128 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294173 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294194 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294348 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294315 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294369 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294453 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294478 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294476 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294492 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xjv\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.294587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.294535 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396098 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.395959 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396100 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396145 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396169 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396293 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396234 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396301 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396340 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396541 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396372 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396718 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396745 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.396798 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396779 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.397143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396823 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xjv\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.397143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.396881 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.397319 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.397291 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.398673 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.398644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.398883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.398866 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.451305 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.451266 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.451628 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.451603 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xjv\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\") pod \"e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:18.729460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:18.729359 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:19.160836 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.160796 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:52:19.161602 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:52:19.161575 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b82b7e6_340b_48c2_a9c7_4f5dee5cbe9e.slice/crio-88e34ca1df5eeece5d8c7fa7dfe4d7bb6ee49a6f69ba25ef515bdc7c4adbffba WatchSource:0}: Error finding container 88e34ca1df5eeece5d8c7fa7dfe4d7bb6ee49a6f69ba25ef515bdc7c4adbffba: Status 404 returned error can't find the container with id 88e34ca1df5eeece5d8c7fa7dfe4d7bb6ee49a6f69ba25ef515bdc7c4adbffba Nov 04 01:52:19.163904 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.163872 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:19.164024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.163950 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:19.164024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.164010 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:19.968290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.968247 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerStarted","Data":"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34"} Nov 04 01:52:19.968476 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:19.968313 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerStarted","Data":"88e34ca1df5eeece5d8c7fa7dfe4d7bb6ee49a6f69ba25ef515bdc7c4adbffba"} Nov 04 01:52:20.177263 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:20.177199 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podStartSLOduration=2.177181814 podStartE2EDuration="2.177181814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:52:20.128539654 +0000 UTC m=+3308.887201815" watchObservedRunningTime="2026-11-04 01:52:20.177181814 +0000 UTC m=+3308.935843974" Nov 04 01:52:20.729880 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:20.729840 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:20.735285 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:20.735258 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:20.975173 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:20.975138 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:20.976146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:20.976122 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:21.581870 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.581785 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:52:21.699097 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.698198 4486 scope.go:122] "RemoveContainer" containerID="d9b15e32068ca72b32d4ef4e8ebca07c5513bc33acaab372d35ad19a22ca71c0" Nov 04 01:52:21.736592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736552 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.736814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736606 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.736814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736658 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.736814 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736689 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.737486 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.737460 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.737687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.737666 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.737923 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.737903 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.738073 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736937 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:52:21.738146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.736975 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:52:21.738146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.737349 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:52:21.738146 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.737944 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:52:21.738376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.738355 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.738555 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.738537 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\") pod \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\" (UID: \"260dabb7-90dc-4fd0-9792-6cc0dccf1be9\") " Nov 04 01:52:21.739024 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.738913 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:52:21.739510 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.739481 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.739643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.739620 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.739643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.739644 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.739846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.739660 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.739846 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.739675 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.741577 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.741540 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "kube-api-access-rxwl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:52:21.741815 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.741788 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:52:21.741933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.741846 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:52:21.742424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.742393 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token" pod "260dabb7-90dc-4fd0-9792-6cc0dccf1be9" (UID: "260dabb7-90dc-4fd0-9792-6cc0dccf1be9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:52:21.841001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.840892 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxwl4\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-kube-api-access-rxwl4\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.841001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.840927 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.841001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.840938 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.841001 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.840951 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/260dabb7-90dc-4fd0-9792-6cc0dccf1be9-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:52:21.983051 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.982974 4486 generic.go:356] "Generic (PLEG): container finished" podID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerID="b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2" exitCode=0 Nov 04 01:52:21.983232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.983070 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" Nov 04 01:52:21.983232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.983103 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerDied","Data":"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2"} Nov 04 01:52:21.983232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.983167 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv" event={"ID":"260dabb7-90dc-4fd0-9792-6cc0dccf1be9","Type":"ContainerDied","Data":"8b2f4561dd49816cf18d6eb84d0de07047ef7894deda11673dbe1e499d8b95f5"} Nov 04 01:52:21.983232 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.983195 4486 scope.go:122] "RemoveContainer" containerID="b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2" Nov 04 01:52:21.993911 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:21.993889 4486 scope.go:122] "RemoveContainer" containerID="a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6" Nov 04 01:52:22.003149 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.003120 4486 scope.go:122] "RemoveContainer" containerID="b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2" Nov 04 01:52:22.003404 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:52:22.003383 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2\": container with ID starting with b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2 not found: ID does not exist" containerID="b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2" Nov 04 01:52:22.003463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.003415 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2"} err="failed to get container status \"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2\": rpc error: code = NotFound desc = could not find container \"b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2\": container with ID starting with b6ab01dc7823b005083bf746bd5c0598b46b6d78089c8736f372d7f8b0e2b6f2 not found: ID does not exist" Nov 04 01:52:22.003463 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.003434 4486 scope.go:122] "RemoveContainer" containerID="a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6" Nov 04 01:52:22.003636 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:52:22.003620 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6\": container with ID starting with a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6 not found: ID does not exist" containerID="a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6" Nov 04 01:52:22.003676 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.003642 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6"} err="failed to get container status \"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6\": rpc error: code = NotFound desc = could not find container \"a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6\": container with ID starting with a77bad274b18f1ed66b7f4e2c2c545f484007ce23106ecb8c1f3fb112677ded6 not found: ID does not exist" Nov 04 01:52:22.141123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.141031 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:52:22.188586 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:22.188549 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4955ec16-openshift-default-74cb7d7d7c-rg5sv"] Nov 04 01:52:23.348967 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:23.348927 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" path="/var/lib/kubelet/pods/260dabb7-90dc-4fd0-9792-6cc0dccf1be9/volumes" Nov 04 01:52:43.836817 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:52:43.836759 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:52:46.137829 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.137789 4486 generic.go:356] "Generic (PLEG): container finished" podID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerID="24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34" exitCode=0 Nov 04 01:52:46.138496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.137870 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerDied","Data":"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34"} Nov 04 01:52:46.138496 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.138292 4486 scope.go:122] "RemoveContainer" containerID="24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34" Nov 04 01:52:46.139559 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.139533 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:46.139640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.139603 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:46.139640 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:46.139634 4486 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 04 01:52:47.162436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:47.162385 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerStarted","Data":"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55"} Nov 04 01:52:47.739009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:47.733167 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:47.739009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:47.734680 4486 patch_prober.go:29] interesting pod/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Nov 04 01:52:47.739009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:47.734736 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Nov 04 01:52:48.732489 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:48.732446 4486 patch_prober.go:29] interesting pod/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Nov 04 01:52:48.733145 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:48.733111 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Nov 04 01:52:49.745167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:49.745127 4486 patch_prober.go:29] interesting pod/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Nov 04 01:52:49.745749 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:49.745716 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Nov 04 01:52:50.732508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:50.732464 4486 patch_prober.go:29] interesting pod/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Nov 04 01:52:50.732840 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:50.732802 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Nov 04 01:52:50.975484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:50.975439 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:51.744009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:51.737483 4486 patch_prober.go:29] interesting pod/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body= Nov 04 01:52:51.744009 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:51.737552 4486 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" Nov 04 01:52:52.734216 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:52.734184 4486 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:52:52.735175 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:52:52.735153 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:53:11.044269 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:53:11.044226 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl"] Nov 04 01:53:11.092085 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:53:11.092041 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3c5056-8gxgl"] Nov 04 01:53:11.348735 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:53:11.348698 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2bcbe1-d138-40c3-bf36-58c3e04cb026" path="/var/lib/kubelet/pods/bb2bcbe1-d138-40c3-bf36-58c3e04cb026/volumes" Nov 04 01:53:13.831879 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:53:13.831839 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:53:21.725215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:53:21.725168 4486 scope.go:122] "RemoveContainer" containerID="07b462ec8e72e0ee501ef1c693c0d94642386da3f91b1ebc27f1986a63cfc257" Nov 04 01:53:43.831539 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:53:43.831498 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:54:13.541855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.541817 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx"] Nov 04 01:54:13.542379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.542306 4486 cpu_manager.go:414] "containerMap: removing container" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:54:13.542379 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.542322 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:54:13.542473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.542400 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:54:13.545643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.545620 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.599074 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.599041 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx"] Nov 04 01:54:13.643101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.643050 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:54:13.643290 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.643230 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pm2xk\"" Nov 04 01:54:13.648273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.648247 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q254k\" (UniqueName: \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.648420 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.648279 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.648484 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.648467 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.750034 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.749959 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.750215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.750059 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q254k\" (UniqueName: \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.750215 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.750084 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.750644 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.750618 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.752452 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.752431 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.804897 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.804821 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q254k\" (UniqueName: \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:13.831545 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:13.831509 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:54:13.859244 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:13.859202 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:14.234436 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:14.234401 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx"] Nov 04 01:54:14.236057 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:54:14.236029 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732f41ea_c9eb_45a0_852e_8778c149e958.slice/crio-21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a WatchSource:0}: Error finding container 21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a: Status 404 returned error can't find the container with id 21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a Nov 04 01:54:14.684719 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:14.684675 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" event={"ID":"732f41ea-c9eb-45a0-852e-8778c149e958","Type":"ContainerStarted","Data":"930329e61ac67f70be83ddf1383b312704a2ef6aa9b0d89215b9778ef2930009"} Nov 04 01:54:14.685174 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:14.684742 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" event={"ID":"732f41ea-c9eb-45a0-852e-8778c149e958","Type":"ContainerStarted","Data":"21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a"} Nov 04 01:54:14.795362 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:14.795255 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" podStartSLOduration=1.7952412629999999 podStartE2EDuration="1.795241263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:54:14.793750263 +0000 UTC m=+3423.552412427" watchObservedRunningTime="2026-11-04 01:54:14.795241263 +0000 UTC m=+3423.553903423" Nov 04 01:54:15.692529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:15.692493 4486 generic.go:356] "Generic (PLEG): container finished" podID="732f41ea-c9eb-45a0-852e-8778c149e958" containerID="930329e61ac67f70be83ddf1383b312704a2ef6aa9b0d89215b9778ef2930009" exitCode=0 Nov 04 01:54:15.692944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:15.692577 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" event={"ID":"732f41ea-c9eb-45a0-852e-8778c149e958","Type":"ContainerDied","Data":"930329e61ac67f70be83ddf1383b312704a2ef6aa9b0d89215b9778ef2930009"} Nov 04 01:54:16.839740 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.839711 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:16.884292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.884248 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\" (UniqueName: \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\") pod \"732f41ea-c9eb-45a0-852e-8778c149e958\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " Nov 04 01:54:16.884502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.884412 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\") pod \"732f41ea-c9eb-45a0-852e-8778c149e958\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " Nov 04 01:54:16.884502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.884476 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\") pod \"732f41ea-c9eb-45a0-852e-8778c149e958\" (UID: \"732f41ea-c9eb-45a0-852e-8778c149e958\") " Nov 04 01:54:16.884874 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.884848 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca" pod "732f41ea-c9eb-45a0-852e-8778c149e958" (UID: "732f41ea-c9eb-45a0-852e-8778c149e958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:54:16.886460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.886434 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca" pod "732f41ea-c9eb-45a0-852e-8778c149e958" (UID: "732f41ea-c9eb-45a0-852e-8778c149e958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:54:16.886460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.886455 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k" pod "732f41ea-c9eb-45a0-852e-8778c149e958" (UID: "732f41ea-c9eb-45a0-852e-8778c149e958"). InnerVolumeSpecName "kube-api-access-q254k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:16.986262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.986170 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:16.986262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.986208 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/732f41ea-c9eb-45a0-852e-8778c149e958-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:16.986262 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:16.986219 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q254k\" (UniqueName: \"kubernetes.io/projected/732f41ea-c9eb-45a0-852e-8778c149e958-kube-api-access-q254k\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:17.378951 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:17.378910 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732f41ea_c9eb_45a0_852e_8778c149e958.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:54:17.379257 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:17.379007 4486 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732f41ea_c9eb_45a0_852e_8778c149e958.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:54:17.705119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:17.705039 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" Nov 04 01:54:17.705119 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:17.705090 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx" event={"ID":"732f41ea-c9eb-45a0-852e-8778c149e958","Type":"ContainerDied","Data":"21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a"} Nov 04 01:54:17.705318 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:17.705140 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21136246737fb656e52953bad48609cfcabc1bfce490c97f4016cd6621ff0f7a" Nov 04 01:54:21.755123 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:21.755091 4486 scope.go:122] "RemoveContainer" containerID="be7b0777419dcfbaa38d84af9fdd15561cce346db36ec63e409da5c559236c46" Nov 04 01:54:21.766610 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:21.766582 4486 scope.go:122] "RemoveContainer" containerID="831570df876a1546e905ff505430b5ed465d7baca67474f4515cf3f48baf86fa" Nov 04 01:54:35.620199 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:35.620158 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx"] Nov 04 01:54:35.667204 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:35.667163 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-3e5e0c-nfscx"] Nov 04 01:54:37.348862 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:37.348822 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732f41ea-c9eb-45a0-852e-8778c149e958" path="/var/lib/kubelet/pods/732f41ea-c9eb-45a0-852e-8778c149e958/volumes" Nov 04 01:54:42.237696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:42.237658 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:54:42.238249 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:42.237876 4486 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" containerID="cri-o://dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55" gracePeriod=30 Nov 04 01:54:43.831703 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:43.831661 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:54:47.501242 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.501176 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:54:47.609761 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609720 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.609979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609783 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.609979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609805 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.609979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609836 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.609979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609860 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.609979 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.609920 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.610316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610063 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.610316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610110 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.610316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610138 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\") pod \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\" (UID: \"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e\") " Nov 04 01:54:47.610316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610214 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:54:47.610316 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610235 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:54:47.610597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610321 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:54:47.610597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610435 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:54:47.610597 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610456 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:54:47.610777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610663 4486 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-certs\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.610777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610686 4486 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-workload-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.610777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610701 4486 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istiod-ca-cert\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.610777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610714 4486 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-data\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.610777 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.610729 4486 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-credential-socket\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.612228 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.612201 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:54:47.612361 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.612312 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:54:47.612625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.612605 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "kube-api-access-v2xjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:47.612625 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.612614 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token" pod "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" (UID: "6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:54:47.712502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.712457 4486 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-podinfo\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.712502 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.712500 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xjv\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-kube-api-access-v2xjv\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.712771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.712526 4486 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-envoy\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.712771 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.712541 4486 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e-istio-token\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:54:47.888409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.888363 4486 generic.go:356] "Generic (PLEG): container finished" podID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerID="dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55" exitCode=0 Nov 04 01:54:47.888409 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.888411 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerDied","Data":"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55"} Nov 04 01:54:47.888803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.888431 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" Nov 04 01:54:47.888803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.888457 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v" event={"ID":"6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e","Type":"ContainerDied","Data":"88e34ca1df5eeece5d8c7fa7dfe4d7bb6ee49a6f69ba25ef515bdc7c4adbffba"} Nov 04 01:54:47.888803 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.888480 4486 scope.go:122] "RemoveContainer" containerID="dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55" Nov 04 01:54:47.898518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.898496 4486 scope.go:122] "RemoveContainer" containerID="24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34" Nov 04 01:54:47.908844 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.908818 4486 scope.go:122] "RemoveContainer" containerID="dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55" Nov 04 01:54:47.909155 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:47.909133 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55\": container with ID starting with dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55 not found: ID does not exist" containerID="dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55" Nov 04 01:54:47.909254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.909167 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55"} err="failed to get container status \"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55\": rpc error: code = NotFound desc = could not find container \"dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55\": container with ID starting with dd5519fd95b66a7ca10ab4a6855cafdaf9fd3ebf9021c846736977933a7b5f55 not found: ID does not exist" Nov 04 01:54:47.909254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.909194 4486 scope.go:122] "RemoveContainer" containerID="24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34" Nov 04 01:54:47.909480 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:54:47.909456 4486 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34\": container with ID starting with 24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34 not found: ID does not exist" containerID="24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34" Nov 04 01:54:47.909529 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:47.909486 4486 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34"} err="failed to get container status \"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34\": rpc error: code = NotFound desc = could not find container \"24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34\": container with ID starting with 24799706032a1ca2bcce2e03f2e0ac7ac7b779e688344e38fa4253b1b63a2a34 not found: ID does not exist" Nov 04 01:54:48.046286 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:48.046249 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:54:48.049536 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:48.049509 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-3e5e0c-openshift-default-78c97578f7-tj49v"] Nov 04 01:54:49.348831 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:49.348795 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" path="/var/lib/kubelet/pods/6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e/volumes" Nov 04 01:54:54.678243 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:54.678126 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4wlbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:54:54.679068 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:54.679015 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4wlbq" podUID="8dd2b86c-c6f6-4156-ba26-5b1116526464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:54:54.858390 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:54.858353 4486 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-cdh5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:54:54.858693 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:54.858659 4486 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-cdh5l" podUID="2a49baee-3767-4b1b-b167-da380781cbaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:54:59.043060 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:59.043021 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r"] Nov 04 01:54:59.091910 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:59.091871 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-t6l9r"] Nov 04 01:54:59.348710 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:54:59.348665 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84331de9-69c6-4e96-88ed-ae88ec782034" path="/var/lib/kubelet/pods/84331de9-69c6-4e96-88ed-ae88ec782034/volumes" Nov 04 01:55:12.763333 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763292 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2"] Nov 04 01:55:12.763768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763729 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.763768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763742 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.763768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763758 4486 cpu_manager.go:414] "containerMap: removing container" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:55:12.763768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763764 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763770 4486 cpu_manager.go:414] "containerMap: removing container" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763776 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763797 4486 cpu_manager.go:414] "containerMap: removing container" podUID="732f41ea-c9eb-45a0-852e-8778c149e958" containerName="revoke-keys" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763801 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732f41ea-c9eb-45a0-852e-8778c149e958" containerName="revoke-keys" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763880 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="732f41ea-c9eb-45a0-852e-8778c149e958" containerName="revoke-keys" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763890 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="260dabb7-90dc-4fd0-9792-6cc0dccf1be9" containerName="istio-proxy" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763900 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.763913 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.763907 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b82b7e6-340b-48c2-a9c7-4f5dee5cbe9e" containerName="istio-proxy" Nov 04 01:55:12.768935 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.768911 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:12.821312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.821273 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2"] Nov 04 01:55:12.865013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.864966 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pm2xk\"" Nov 04 01:55:12.865013 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.864997 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:55:12.936326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.936282 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmsn2\" (UniqueName: \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:12.936326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.936329 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:12.936587 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:12.936493 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.037482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.037373 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmsn2\" (UniqueName: \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.037482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.037426 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.037482 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.037469 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.038160 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.038134 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.039963 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.039932 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.092818 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.092782 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmsn2\" (UniqueName: \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\") pod \"maas-api-revoke-sub-f03629db-bqfh2\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.382504 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.382471 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:13.750430 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:13.750399 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2"] Nov 04 01:55:13.752101 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:55:13.752069 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135616cc_b387_43ed_9691_86f429e69b04.slice/crio-b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38 WatchSource:0}: Error finding container b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38: Status 404 returned error can't find the container with id b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38 Nov 04 01:55:13.833036 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:55:13.832994 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:55:14.149065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:14.149018 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" event={"ID":"135616cc-b387-43ed-9691-86f429e69b04","Type":"ContainerStarted","Data":"ecf93354616ff3fe6ca343fa104db76a035a0c2ca20b1d420c86942fbb7f3b02"} Nov 04 01:55:14.149273 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:14.149084 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" event={"ID":"135616cc-b387-43ed-9691-86f429e69b04","Type":"ContainerStarted","Data":"b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38"} Nov 04 01:55:14.260308 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:14.260248 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" podStartSLOduration=2.260234956 podStartE2EDuration="2.260234956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:55:14.258204628 +0000 UTC m=+3483.016866788" watchObservedRunningTime="2026-11-04 01:55:14.260234956 +0000 UTC m=+3483.018897154" Nov 04 01:55:15.157585 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:15.157545 4486 generic.go:356] "Generic (PLEG): container finished" podID="135616cc-b387-43ed-9691-86f429e69b04" containerID="ecf93354616ff3fe6ca343fa104db76a035a0c2ca20b1d420c86942fbb7f3b02" exitCode=0 Nov 04 01:55:15.158069 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:15.157640 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" event={"ID":"135616cc-b387-43ed-9691-86f429e69b04","Type":"ContainerDied","Data":"ecf93354616ff3fe6ca343fa104db76a035a0c2ca20b1d420c86942fbb7f3b02"} Nov 04 01:55:16.297080 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.297045 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:16.475167 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.475070 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\" (UniqueName: \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\") pod \"135616cc-b387-43ed-9691-86f429e69b04\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " Nov 04 01:55:16.475352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.475178 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\") pod \"135616cc-b387-43ed-9691-86f429e69b04\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " Nov 04 01:55:16.475352 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.475223 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\") pod \"135616cc-b387-43ed-9691-86f429e69b04\" (UID: \"135616cc-b387-43ed-9691-86f429e69b04\") " Nov 04 01:55:16.475638 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.475606 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca" pod "135616cc-b387-43ed-9691-86f429e69b04" (UID: "135616cc-b387-43ed-9691-86f429e69b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:55:16.477394 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.477364 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca" pod "135616cc-b387-43ed-9691-86f429e69b04" (UID: "135616cc-b387-43ed-9691-86f429e69b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:55:16.477518 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.477501 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2" pod "135616cc-b387-43ed-9691-86f429e69b04" (UID: "135616cc-b387-43ed-9691-86f429e69b04"). InnerVolumeSpecName "kube-api-access-qmsn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:55:16.576094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.576047 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135616cc-b387-43ed-9691-86f429e69b04-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:16.576094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.576087 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135616cc-b387-43ed-9691-86f429e69b04-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:16.576094 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:16.576098 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmsn2\" (UniqueName: \"kubernetes.io/projected/135616cc-b387-43ed-9691-86f429e69b04-kube-api-access-qmsn2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:17.171583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:17.171550 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" Nov 04 01:55:17.171583 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:17.171562 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-bqfh2" event={"ID":"135616cc-b387-43ed-9691-86f429e69b04","Type":"ContainerDied","Data":"b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38"} Nov 04 01:55:17.171876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:17.171614 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b210829a3e120ff8c8a51fbc037afff9bed836b564fca726039ba676aee3ab38" Nov 04 01:55:31.043657 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:31.043620 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q"] Nov 04 01:55:31.092857 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:31.092812 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-whq2q"] Nov 04 01:55:31.364853 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:31.364816 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab7397f-f5c5-40e0-a5bd-13268a15a0ba" path="/var/lib/kubelet/pods/5ab7397f-f5c5-40e0-a5bd-13268a15a0ba/volumes" Nov 04 01:55:33.661102 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.661067 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj"] Nov 04 01:55:33.661516 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.661503 4486 cpu_manager.go:414] "containerMap: removing container" podUID="135616cc-b387-43ed-9691-86f429e69b04" containerName="revoke-keys" Nov 04 01:55:33.661561 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.661516 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135616cc-b387-43ed-9691-86f429e69b04" containerName="revoke-keys" Nov 04 01:55:33.661608 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.661599 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="135616cc-b387-43ed-9691-86f429e69b04" containerName="revoke-keys" Nov 04 01:55:33.665101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.665071 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.763729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.763693 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:55:33.763729 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.763694 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pm2xk\"" Nov 04 01:55:33.768180 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.768148 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj"] Nov 04 01:55:33.798086 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.798042 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.798268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.798204 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsk2\" (UniqueName: \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.798268 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.798244 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.899876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.899843 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsk2\" (UniqueName: \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.900100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.899886 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.900100 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.899952 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.900662 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.900639 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.902429 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.902401 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.954195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.954125 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsk2\" (UniqueName: \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\") pod \"maas-api-revoke-sub-0165d7d2-fr8sj\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:33.978694 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:33.978658 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:34.346422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:34.346383 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj"] Nov 04 01:55:34.347601 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:55:34.347568 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33976691_27ff_40c1_84cb_1a1c593edb22.slice/crio-133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c WatchSource:0}: Error finding container 133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c: Status 404 returned error can't find the container with id 133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c Nov 04 01:55:35.307533 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:35.307496 4486 generic.go:356] "Generic (PLEG): container finished" podID="33976691-27ff-40c1-84cb-1a1c593edb22" containerID="e60c540a0561b2b82a6109a6a56b348c423cd14c1777ec54b65dbc5138679d8d" exitCode=0 Nov 04 01:55:35.308065 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:35.307548 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" event={"ID":"33976691-27ff-40c1-84cb-1a1c593edb22","Type":"ContainerDied","Data":"e60c540a0561b2b82a6109a6a56b348c423cd14c1777ec54b65dbc5138679d8d"} Nov 04 01:55:35.308195 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:35.308097 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" event={"ID":"33976691-27ff-40c1-84cb-1a1c593edb22","Type":"ContainerStarted","Data":"133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c"} Nov 04 01:55:36.456928 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.456899 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:36.627618 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.627522 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\") pod \"33976691-27ff-40c1-84cb-1a1c593edb22\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " Nov 04 01:55:36.627807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.627679 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\" (UniqueName: \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\") pod \"33976691-27ff-40c1-84cb-1a1c593edb22\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " Nov 04 01:55:36.627807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.627759 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\") pod \"33976691-27ff-40c1-84cb-1a1c593edb22\" (UID: \"33976691-27ff-40c1-84cb-1a1c593edb22\") " Nov 04 01:55:36.627944 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.627911 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca" pod "33976691-27ff-40c1-84cb-1a1c593edb22" (UID: "33976691-27ff-40c1-84cb-1a1c593edb22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:55:36.628157 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.628140 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:36.629895 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.629866 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca" pod "33976691-27ff-40c1-84cb-1a1c593edb22" (UID: "33976691-27ff-40c1-84cb-1a1c593edb22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:55:36.629964 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.629920 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2" pod "33976691-27ff-40c1-84cb-1a1c593edb22" (UID: "33976691-27ff-40c1-84cb-1a1c593edb22"). InnerVolumeSpecName "kube-api-access-xpsk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:55:36.729743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.729689 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33976691-27ff-40c1-84cb-1a1c593edb22-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:36.729743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:36.729735 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpsk2\" (UniqueName: \"kubernetes.io/projected/33976691-27ff-40c1-84cb-1a1c593edb22-kube-api-access-xpsk2\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:37.321312 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:37.321275 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" event={"ID":"33976691-27ff-40c1-84cb-1a1c593edb22","Type":"ContainerDied","Data":"133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c"} Nov 04 01:55:37.321500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:37.321352 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133b89821749a1e13b2367ac825515ba93d5b9ac1b0cad4f8ba1499cd100340c" Nov 04 01:55:37.321500 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:37.321291 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fr8sj" Nov 04 01:55:43.834894 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:55:43.834849 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:55:45.334958 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.334923 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm"] Nov 04 01:55:45.335414 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.335391 4486 cpu_manager.go:414] "containerMap: removing container" podUID="33976691-27ff-40c1-84cb-1a1c593edb22" containerName="revoke-keys" Nov 04 01:55:45.335460 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.335414 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33976691-27ff-40c1-84cb-1a1c593edb22" containerName="revoke-keys" Nov 04 01:55:45.335498 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.335490 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="33976691-27ff-40c1-84cb-1a1c593edb22" containerName="revoke-keys" Nov 04 01:55:45.338812 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.338785 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.392804 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.392713 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm"] Nov 04 01:55:45.415649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.415595 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.415824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.415729 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.415824 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.415786 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msx64\" (UniqueName: \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.436046 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.436007 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:55:45.436237 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.436137 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pm2xk\"" Nov 04 01:55:45.516598 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.516555 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.516799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.516622 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.516799 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.516662 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msx64\" (UniqueName: \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.517538 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.517504 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.519154 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.519125 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.571517 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.571476 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msx64\" (UniqueName: \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\") pod \"maas-api-revoke-sub-6b19338f-ph8dm\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:45.652961 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:45.652864 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:46.023018 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:46.022975 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm"] Nov 04 01:55:46.025213 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:55:46.025183 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7e658b_feaf_4604_bed6_52603fc1a579.slice/crio-eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509 WatchSource:0}: Error finding container eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509: Status 404 returned error can't find the container with id eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509 Nov 04 01:55:46.375823 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:46.375778 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" event={"ID":"dd7e658b-feaf-4604-bed6-52603fc1a579","Type":"ContainerStarted","Data":"949757c7729e6546701f4bc5231eebfcba725d00c6bfecea43f9b290c6030cce"} Nov 04 01:55:46.376328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:46.375837 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" event={"ID":"dd7e658b-feaf-4604-bed6-52603fc1a579","Type":"ContainerStarted","Data":"eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509"} Nov 04 01:55:46.488143 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:46.488085 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" podStartSLOduration=1.488070139 podStartE2EDuration="1.488070139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:55:46.485493749 +0000 UTC m=+3515.244155910" watchObservedRunningTime="2026-11-04 01:55:46.488070139 +0000 UTC m=+3515.246732297" Nov 04 01:55:47.382901 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:47.382863 4486 generic.go:356] "Generic (PLEG): container finished" podID="dd7e658b-feaf-4604-bed6-52603fc1a579" containerID="949757c7729e6546701f4bc5231eebfcba725d00c6bfecea43f9b290c6030cce" exitCode=0 Nov 04 01:55:47.383393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:47.382955 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" event={"ID":"dd7e658b-feaf-4604-bed6-52603fc1a579","Type":"ContainerDied","Data":"949757c7729e6546701f4bc5231eebfcba725d00c6bfecea43f9b290c6030cce"} Nov 04 01:55:48.529077 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.529046 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:48.549221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.549186 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\" (UniqueName: \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\") pod \"dd7e658b-feaf-4604-bed6-52603fc1a579\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " Nov 04 01:55:48.549413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.549238 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\") pod \"dd7e658b-feaf-4604-bed6-52603fc1a579\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " Nov 04 01:55:48.549413 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.549352 4486 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\") pod \"dd7e658b-feaf-4604-bed6-52603fc1a579\" (UID: \"dd7e658b-feaf-4604-bed6-52603fc1a579\") " Nov 04 01:55:48.549687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.549660 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca" pod "dd7e658b-feaf-4604-bed6-52603fc1a579" (UID: "dd7e658b-feaf-4604-bed6-52603fc1a579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:55:48.551473 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.551431 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64" pod "dd7e658b-feaf-4604-bed6-52603fc1a579" (UID: "dd7e658b-feaf-4604-bed6-52603fc1a579"). InnerVolumeSpecName "kube-api-access-msx64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:55:48.551596 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.551502 4486 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca" pod "dd7e658b-feaf-4604-bed6-52603fc1a579" (UID: "dd7e658b-feaf-4604-bed6-52603fc1a579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:55:48.650772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.650692 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-xks-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:48.650772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.650734 4486 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msx64\" (UniqueName: \"kubernetes.io/projected/dd7e658b-feaf-4604-bed6-52603fc1a579-kube-api-access-msx64\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:48.650772 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:48.650750 4486 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e658b-feaf-4604-bed6-52603fc1a579-maas-api-openshift-ca\") on node \"ip-10-0-129-114.us-west-2.compute.internal\" DevicePath \"\"" Nov 04 01:55:49.396474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:49.396438 4486 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" Nov 04 01:55:49.396474 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:49.396467 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ph8dm" event={"ID":"dd7e658b-feaf-4604-bed6-52603fc1a579","Type":"ContainerDied","Data":"eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509"} Nov 04 01:55:49.396743 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:49.396511 4486 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba2a3824b70b13610b9b51d5f7426bf2e052e6907c3fe60512df8a4a213e509" Nov 04 01:55:57.432294 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:55:57.432258 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59994988cf-d8rwg_de8185b1-5c0e-420c-9176-c907147bb322/authorino/0.log" Nov 04 01:56:04.045509 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:04.045471 4486 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm"] Nov 04 01:56:04.092933 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:04.092899 4486 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca319fb-9c6jm"] Nov 04 01:56:05.349643 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:05.349608 4486 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a653897-36f7-47f0-94eb-858b50a55dc7" path="/var/lib/kubelet/pods/3a653897-36f7-47f0-94eb-858b50a55dc7/volumes" Nov 04 01:56:08.252377 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:08.252339 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cc77699f-8pjtq_9ab844e2-731c-417c-8630-09f10b5f05b3/manager/0.log" Nov 04 01:56:08.646023 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:08.645978 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6x8gq_35775c31-4459-4fe9-beda-839483a09ef4/manager/0.log" Nov 04 01:56:08.889324 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:08.889284 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwp88_33067ffe-3e40-4d78-9fa2-9c8e8c7617b5/manager/0.log" Nov 04 01:56:10.657807 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:10.657772 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-t2z7h_14711a91-e9be-49f9-a6cd-8acc642a1af5/proxy/0.log" Nov 04 01:56:11.998877 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:11.998843 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fr8sj_33976691-27ff-40c1-84cb-1a1c593edb22/revoke-keys/0.log" Nov 04 01:56:12.218876 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:12.218849 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ph8dm_dd7e658b-feaf-4604-bed6-52603fc1a579/revoke-keys/0.log" Nov 04 01:56:12.433326 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:12.433291 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-bqfh2_135616cc-b387-43ed-9691-86f429e69b04/revoke-keys/0.log" Nov 04 01:56:12.671700 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:12.671664 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b46jx_dc438b0b-8727-463a-ac5e-49b0c22ea2d5/postgres/0.log" Nov 04 01:56:13.832230 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:56:13.832188 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:56:14.222976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:14.222882 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59994988cf-d8rwg_de8185b1-5c0e-420c-9176-c907147bb322/authorino/0.log" Nov 04 01:56:14.694604 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:14.694571 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tvh5l_130229ab-0216-44e2-aca7-7affd622750e/manager/0.log" Nov 04 01:56:14.913681 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:14.913645 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ndrjb_88544e2b-965f-4f08-9dd0-dc80129ccecc/manager/0.log" Nov 04 01:56:15.387437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:15.387407 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pt7b_fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3/manager/0.log" Nov 04 01:56:15.976581 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:15.976545 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qt25f_45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7/manager/0.log" Nov 04 01:56:16.798444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:16.798416 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm_b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b/istio-proxy/0.log" Nov 04 01:56:18.250826 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:18.250793 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-4wlbq_8dd2b86c-c6f6-4156-ba26-5b1116526464/istio-proxy/0.log" Nov 04 01:56:18.539682 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:18.539599 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-cdh5l_2a49baee-3767-4b1b-b167-da380781cbaa/istio-proxy/0.log" Nov 04 01:56:19.041976 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:19.041939 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dkxqn_4fc7f914-cc9b-4ead-9299-822ce680bc33/istio-proxy/0.log" Nov 04 01:56:20.273344 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:20.273309 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv_6536c33b-bd9a-4f7b-b963-985785fa85ff/istio-proxy/0.log" Nov 04 01:56:20.514343 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:20.514301 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v7hk2_ab5b3264-5961-4fc2-be5d-5c68426bb887/payload-pre-processing/0.log" Nov 04 01:56:20.742783 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:20.742748 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:56:21.794835 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:21.794802 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7_fec97081-9922-47be-8eb7-3d25b8500e75/main/0.log" Nov 04 01:56:21.808916 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:21.808883 4486 scope.go:122] "RemoveContainer" containerID="137c351879b8f2f884d90305b5a2fe1351051d77b38ab84a1f818565db7eab56" Nov 04 01:56:22.324447 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:22.324413 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8vwvq_4bed0a38-3d09-483e-8c64-2bc858586a46/main/0.log" Nov 04 01:56:23.082875 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:23.082793 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-zrnhb_344cadfc-a8d8-4ced-88d9-c046e7ecd3e9/main/0.log" Nov 04 01:56:23.314948 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:23.314913 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8_c0d70473-bec3-4c76-95c2-7227ed44617b/main/0.log" Nov 04 01:56:24.513444 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:24.513407 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59994988cf-d8rwg_de8185b1-5c0e-420c-9176-c907147bb322/authorino/0.log" Nov 04 01:56:33.789426 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:33.789378 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cc77699f-8pjtq_9ab844e2-731c-417c-8630-09f10b5f05b3/manager/0.log" Nov 04 01:56:34.174573 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:34.174534 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6x8gq_35775c31-4459-4fe9-beda-839483a09ef4/manager/0.log" Nov 04 01:56:34.418769 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:34.418727 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-mwp88_33067ffe-3e40-4d78-9fa2-9c8e8c7617b5/manager/0.log" Nov 04 01:56:35.887422 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:35.887386 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-t2z7h_14711a91-e9be-49f9-a6cd-8acc642a1af5/proxy/0.log" Nov 04 01:56:37.003221 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:37.003184 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fr8sj_33976691-27ff-40c1-84cb-1a1c593edb22/revoke-keys/0.log" Nov 04 01:56:37.223101 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:37.223072 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ph8dm_dd7e658b-feaf-4604-bed6-52603fc1a579/revoke-keys/0.log" Nov 04 01:56:37.446092 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:37.446061 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-bqfh2_135616cc-b387-43ed-9691-86f429e69b04/revoke-keys/0.log" Nov 04 01:56:37.670855 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:37.670825 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b46jx_dc438b0b-8727-463a-ac5e-49b0c22ea2d5/postgres/0.log" Nov 04 01:56:38.842490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:38.842456 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59994988cf-d8rwg_de8185b1-5c0e-420c-9176-c907147bb322/authorino/0.log" Nov 04 01:56:39.314733 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:39.314705 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tvh5l_130229ab-0216-44e2-aca7-7affd622750e/manager/0.log" Nov 04 01:56:39.553872 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:39.553838 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ndrjb_88544e2b-965f-4f08-9dd0-dc80129ccecc/manager/0.log" Nov 04 01:56:40.029416 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:40.029383 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pt7b_fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3/manager/0.log" Nov 04 01:56:40.551266 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:40.551230 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qt25f_45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7/manager/0.log" Nov 04 01:56:41.152467 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:41.152439 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm_b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b/istio-proxy/0.log" Nov 04 01:56:42.627784 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:42.627747 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-4wlbq_8dd2b86c-c6f6-4156-ba26-5b1116526464/istio-proxy/0.log" Nov 04 01:56:42.910490 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:42.910406 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-cdh5l_2a49baee-3767-4b1b-b167-da380781cbaa/istio-proxy/0.log" Nov 04 01:56:43.430138 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:43.430098 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-dkxqn_4fc7f914-cc9b-4ead-9299-822ce680bc33/istio-proxy/0.log" Nov 04 01:56:43.831001 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:56:43.830938 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:56:44.590152 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:44.590116 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-2tswv_6536c33b-bd9a-4f7b-b963-985785fa85ff/istio-proxy/0.log" Nov 04 01:56:44.813113 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:44.813083 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-v7hk2_ab5b3264-5961-4fc2-be5d-5c68426bb887/payload-pre-processing/0.log" Nov 04 01:56:45.033063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:45.032946 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:56:45.837271 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:45.837242 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-25jz7_fec97081-9922-47be-8eb7-3d25b8500e75/main/0.log" Nov 04 01:56:46.334937 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:46.334901 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8vwvq_4bed0a38-3d09-483e-8c64-2bc858586a46/main/0.log" Nov 04 01:56:47.003346 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:47.003313 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-zrnhb_344cadfc-a8d8-4ced-88d9-c046e7ecd3e9/main/0.log" Nov 04 01:56:47.242508 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:56:47.242469 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-qdkq8_c0d70473-bec3-4c76-95c2-7227ed44617b/main/0.log" Nov 04 01:57:13.832131 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:57:13.832092 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:57:18.239950 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.239908 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:57:18.255288 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.255258 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:57:18.264393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.264365 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmgw85_29da81f9-6162-4fb5-acd5-06eb5fe72c32/extract/0.log" Nov 04 01:57:18.278371 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.278343 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9dx8l_a7323316-cacf-4ca1-afb1-7b38b203dfd7/ovn-acl-logging/0.log" Nov 04 01:57:18.629337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.629308 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:57:18.855159 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:18.855132 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:57:21.832437 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:21.832402 4486 scope.go:122] "RemoveContainer" containerID="b0c8d0d22465cd0075763c917c0f3151abb9a935c1357c5c4e18b0a91ce19bfa" Nov 04 01:57:21.842376 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:21.842343 4486 scope.go:122] "RemoveContainer" containerID="071be81f5618d82e2b94cc5a7d99f1b2febd5752472a0e0ec3a0f24863eb13b4" Nov 04 01:57:43.832085 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:57:43.832037 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:57:58.322825 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.322742 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:57:58.552860 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.552831 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xzdcs_15448f77-e10f-485b-903a-2c96a15f4460/payload-processing/0.log" Nov 04 01:57:58.924766 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.924732 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7gzq6/must-gather-258km"] Nov 04 01:57:58.925254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.925221 4486 cpu_manager.go:414] "containerMap: removing container" podUID="dd7e658b-feaf-4604-bed6-52603fc1a579" containerName="revoke-keys" Nov 04 01:57:58.925254 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.925252 4486 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7e658b-feaf-4604-bed6-52603fc1a579" containerName="revoke-keys" Nov 04 01:57:58.925471 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.925342 4486 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7e658b-feaf-4604-bed6-52603fc1a579" containerName="revoke-keys" Nov 04 01:57:58.928697 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.928676 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:58.979141 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:58.979111 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gzq6\"/\"kube-root-ca.crt\"" Nov 04 01:57:59.025920 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.025877 4486 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gzq6\"/\"openshift-service-ca.crt\"" Nov 04 01:57:59.026176 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.026009 4486 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7gzq6\"/\"default-dockercfg-pd6hx\"" Nov 04 01:57:59.029137 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.029111 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/must-gather-258km"] Nov 04 01:57:59.092107 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.092063 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqd9l\" (UniqueName: \"kubernetes.io/projected/19a5a781-dd22-4d71-892a-8777bf9fb897-kube-api-access-pqd9l\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.092307 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.092204 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a5a781-dd22-4d71-892a-8777bf9fb897-must-gather-output\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.193292 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.193201 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqd9l\" (UniqueName: \"kubernetes.io/projected/19a5a781-dd22-4d71-892a-8777bf9fb897-kube-api-access-pqd9l\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.193457 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.193302 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a5a781-dd22-4d71-892a-8777bf9fb897-must-gather-output\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.193661 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.193644 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a5a781-dd22-4d71-892a-8777bf9fb897-must-gather-output\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.256528 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.256491 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqd9l\" (UniqueName: \"kubernetes.io/projected/19a5a781-dd22-4d71-892a-8777bf9fb897-kube-api-access-pqd9l\") pod \"must-gather-258km\" (UID: \"19a5a781-dd22-4d71-892a-8777bf9fb897\") " pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.544116 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.544006 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/must-gather-258km" Nov 04 01:57:59.917106 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.917074 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/must-gather-258km"] Nov 04 01:57:59.918716 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:57:59.918688 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a5a781_dd22_4d71_892a_8777bf9fb897.slice/crio-c755f63d0bab5a39bd7e8ffe5aecf9731da8603d8633c323dc3070ca6c3c5efa WatchSource:0}: Error finding container c755f63d0bab5a39bd7e8ffe5aecf9731da8603d8633c323dc3070ca6c3c5efa: Status 404 returned error can't find the container with id c755f63d0bab5a39bd7e8ffe5aecf9731da8603d8633c323dc3070ca6c3c5efa Nov 04 01:57:59.920696 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:57:59.920681 4486 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:58:00.188385 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:00.188299 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/must-gather-258km" event={"ID":"19a5a781-dd22-4d71-892a-8777bf9fb897","Type":"ContainerStarted","Data":"c755f63d0bab5a39bd7e8ffe5aecf9731da8603d8633c323dc3070ca6c3c5efa"} Nov 04 01:58:02.205713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:02.205668 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/must-gather-258km" event={"ID":"19a5a781-dd22-4d71-892a-8777bf9fb897","Type":"ContainerStarted","Data":"f7d0d05fd07e9102eb7a2145c4e65c152bbad7f5fed5b4581d4464318dc816e1"} Nov 04 01:58:02.205713 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:02.205718 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/must-gather-258km" event={"ID":"19a5a781-dd22-4d71-892a-8777bf9fb897","Type":"ContainerStarted","Data":"d3fa717f4e413a7e62169a38eba64843c3f4fe70a11507670f94acc2dc2c6a4b"} Nov 04 01:58:02.315248 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:02.315182 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7gzq6/must-gather-258km" podStartSLOduration=2.494684043 podStartE2EDuration="4.315166721s" totalImagesPullingTime="1.820482678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:58:02.313234177 +0000 UTC m=+3651.071896361" watchObservedRunningTime="2026-11-04 01:58:02.315166721 +0000 UTC m=+3651.073828914" Nov 04 01:58:10.339730 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:10.339695 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lkssf_c12d5c7d-b992-4093-8b4f-1bf058820213/global-pull-secret-syncer/0.log" Nov 04 01:58:13.860864 ip-10-0-129-114 kubenswrapper[4486]: E1104 01:58:13.860792 4486 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nsgcc" containerName="extractor" Nov 04 01:58:14.760000 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:14.759958 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8knts_f9554504-e458-45eb-ac2f-fb8bd160802e/konnectivity-agent/0.log" Nov 04 01:58:15.164894 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:15.164853 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-114.us-west-2.compute.internal_93ea24cc66811a8502960a15b9987f67/haproxy/0.log" Nov 04 01:58:19.146649 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.146393 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59994988cf-d8rwg_de8185b1-5c0e-420c-9176-c907147bb322/authorino/0.log" Nov 04 01:58:19.414706 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.414608 4486 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl"] Nov 04 01:58:19.419949 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.419920 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.476306 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.476236 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl"] Nov 04 01:58:19.516592 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.516560 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-tvh5l_130229ab-0216-44e2-aca7-7affd622750e/manager/0.log" Nov 04 01:58:19.536393 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.536357 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsqd\" (UniqueName: \"kubernetes.io/projected/c7389257-ec07-41c8-a12f-0cfcf576b47b-kube-api-access-5rsqd\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.536736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.536410 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-proc\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.536736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.536517 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-podres\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.536736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.536604 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-lib-modules\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.536736 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.536670 4486 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-sys\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.637794 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637755 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsqd\" (UniqueName: \"kubernetes.io/projected/c7389257-ec07-41c8-a12f-0cfcf576b47b-kube-api-access-5rsqd\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637816 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-proc\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637854 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-podres\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637901 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-lib-modules\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637963 4486 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-sys\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637900 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-proc\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638039 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.637962 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-lib-modules\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.638027 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-podres\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.638337 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.638143 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7389257-ec07-41c8-a12f-0cfcf576b47b-sys\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.696688 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.696611 4486 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsqd\" (UniqueName: \"kubernetes.io/projected/c7389257-ec07-41c8-a12f-0cfcf576b47b-kube-api-access-5rsqd\") pod \"perf-node-gather-daemonset-6j2gl\" (UID: \"c7389257-ec07-41c8-a12f-0cfcf576b47b\") " pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.732063 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.731993 4486 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:19.961544 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:19.961456 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-ndrjb_88544e2b-965f-4f08-9dd0-dc80129ccecc/manager/0.log" Nov 04 01:58:20.101716 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.101688 4486 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl"] Nov 04 01:58:20.102790 ip-10-0-129-114 kubenswrapper[4486]: W1104 01:58:20.102755 4486 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc7389257_ec07_41c8_a12f_0cfcf576b47b.slice/crio-8c39de13420e25f7409fd6b1e5c764d66b6152b2c1543c06a515551ce940ee72 WatchSource:0}: Error finding container 8c39de13420e25f7409fd6b1e5c764d66b6152b2c1543c06a515551ce940ee72: Status 404 returned error can't find the container with id 8c39de13420e25f7409fd6b1e5c764d66b6152b2c1543c06a515551ce940ee72 Nov 04 01:58:20.395424 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.395379 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" event={"ID":"c7389257-ec07-41c8-a12f-0cfcf576b47b","Type":"ContainerStarted","Data":"be33601e19338afe1ed1677723b51a38698494d8fd66df79e0bd7c89d8daaa66"} Nov 04 01:58:20.395898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.395439 4486 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" event={"ID":"c7389257-ec07-41c8-a12f-0cfcf576b47b","Type":"ContainerStarted","Data":"8c39de13420e25f7409fd6b1e5c764d66b6152b2c1543c06a515551ce940ee72"} Nov 04 01:58:20.395898 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.395489 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:20.507336 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.507227 4486 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" podStartSLOduration=1.507210763 podStartE2EDuration="1.507210763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:58:20.504448681 +0000 UTC m=+3669.263110842" watchObservedRunningTime="2026-11-04 01:58:20.507210763 +0000 UTC m=+3669.265872924" Nov 04 01:58:20.891495 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:20.891455 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6pt7b_fc3d7323-ac5b-4ade-bc40-09b0ba4acdf3/manager/0.log" Nov 04 01:58:21.809865 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:21.809815 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-qt25f_45ac63b3-2cc2-4ff4-b7f9-cb03288f56b7/manager/0.log" Nov 04 01:58:21.869883 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:21.869848 4486 scope.go:122] "RemoveContainer" containerID="dd0ffd9eb00ec9b05f64391c35562c53cf8ea7dcb4234e2da5f747c2763e7325" Nov 04 01:58:26.412556 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:26.412525 4486 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7gzq6/perf-node-gather-daemonset-6j2gl" Nov 04 01:58:30.060712 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:30.060672 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-xmdnr_36270d33-26d2-4047-9db9-0d418dda1d02/monitoring-plugin/0.log" Nov 04 01:58:30.511768 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:30.511740 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfb5n_1a6fab7b-b666-4651-a2ef-9b4c5dbadf84/node-exporter/0.log" Nov 04 01:58:30.960627 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:30.960597 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfb5n_1a6fab7b-b666-4651-a2ef-9b4c5dbadf84/kube-rbac-proxy/0.log" Nov 04 01:58:31.406328 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:31.406296 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfb5n_1a6fab7b-b666-4651-a2ef-9b4c5dbadf84/init-textfile/0.log" Nov 04 01:58:38.985704 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:38.985672 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fhcrc_16f4716a-952a-4390-a02d-9628df25ef87/prometheus-operator/0.log" Nov 04 01:58:39.426431 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:39.426389 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-fhcrc_16f4716a-952a-4390-a02d-9628df25ef87/kube-rbac-proxy/0.log" Nov 04 01:58:41.652687 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:41.652652 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68795d64cd-6f466_ef840115-544e-4e5b-b573-9938e28ecec7/thanos-query/0.log" Nov 04 01:58:42.096088 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:42.096039 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68795d64cd-6f466_ef840115-544e-4e5b-b573-9938e28ecec7/kube-rbac-proxy-web/0.log" Nov 04 01:58:42.542354 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:42.542259 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68795d64cd-6f466_ef840115-544e-4e5b-b573-9938e28ecec7/kube-rbac-proxy/0.log" Nov 04 01:58:43.016900 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:43.016870 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68795d64cd-6f466_ef840115-544e-4e5b-b573-9938e28ecec7/prom-label-proxy/0.log" Nov 04 01:58:43.104265 ip-10-0-129-114 kubenswrapper[4486]: I1104 01:58:43.104232 4486 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-jvhhm_b7b5d7d1-6e26-4ca5-8f19-400bc9e5121b/istio-proxy/0.log"